+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit
+ bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20190507/A0c --output-run DEEP/20190507/A0c/science#step0/20240309T173428Z --qgraph pipeline.qgraph
lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process.
lsst.ctrl.bps.drivers INFO: Starting submission process
lsst.ctrl.bps.drivers INFO: Initializing execution environment
lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 27.4728 seconds; current memory usage: 0.178 Gibyte, delta: 0.041 Gibyte, peak delta: 0.053 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph)
lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.4634 seconds
lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.4750 seconds
lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/pipeline_orig.qgraph DEEP/20190507/A0c/science#step0/20240309T173428Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/pipeline.qgraph 
lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 3.8195 seconds
lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 17.1362 seconds; current memory usage: 0.272 Gibyte, delta: 0.094 Gibyte, peak delta: 0.082 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.263 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs)
lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0365 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.263 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 372 cluster(s)
lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow)
lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20190507_A0c_science#step0_20240309T173428Z'
lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.5841 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.263 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 374 job(s) (including final)
lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow)
parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses
    s_addresses.add(address_by_interface(interface))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper
    retval = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface
    return socket.inet_ntoa(fcntl.ioctl(
                            ^^^^^^^^^^^^
OSError: [Errno 99] Cannot assign requested address
lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z
lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 1.7537 seconds; current memory usage: 0.273 Gibyte, delta: 0.001 Gibyte, peak delta: 0.001 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.273 Gibyte (main), 0.263 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting submit stage
lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution
parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000
parsl.dataflow.dflow INFO: Starting DataFlowKernel with config
Config(
    app_cache=True, 
    checkpoint_files=None, 
    checkpoint_mode='task_exit', 
    checkpoint_period=None, 
    executors=[MultiHighThroughputExecutor()], 
    garbage_collect=True, 
    initialize_logging=True, 
    internal_tasks_max_threads=10, 
    max_idletime=120.0, 
    monitoring=None, 
    retries=1, 
    retry_handler=None, 
    run_dir='runinfo', 
    strategy='simple', 
    usage_tracking=False
)
parsl.dataflow.dflow INFO: Parsl version: 2023.06.12
parsl.usage_tracking.usage DEBUG: Tracking status: False
parsl.dataflow.dflow INFO: Run id is: fdd1974a-3438-4116-a542-0dada37d1746
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py
parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name
parsl.dataflow.memoization INFO: App caching initialized
parsl.dataflow.strategy DEBUG: Scaling strategy: simple
parsl.executors.high_throughput.executor DEBUG: Starting queue management thread
parsl.executors.high_throughput.executor DEBUG: queue management worker starting
parsl.executors.high_throughput.executor DEBUG: Started queue management thread
Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 0 executors
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23383474330768
parsl.executors.high_throughput.executor DEBUG: Created management thread: <Thread(HTEX-Queue-Management-Thread, started daemon 23383451301632)>
parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id={block_id} --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider:
<proc_lsst.multi.MultiProvider object at 0x1544685e7b90>
parsl.executors.status_handling INFO: Scaling out by 1 blocks
parsl.executors.status_handling INFO: Allocated block ID 0
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0
2024-03-09 17:35:57 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-0
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-0
2024-03-09 17:35:57 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:35:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-09 17:35:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:35:57 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2 to local
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2 to local
parsl.providers.local.local DEBUG: Launching in remote mode
2024-03-09 17:35:58 proc_lsst.multi:170 [INFO]  [multi] job_id 64340
proc_lsst.multi INFO: [multi] job_id 64340
2024-03-09 17:35:58 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:35:58 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:35:58 proc_lsst.multi:178 [INFO]  [multi] provider local accepted submit and returned 64340
proc_lsst.multi INFO: [multi] provider local accepted submit and returned 64340
parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 64340
parsl.dataflow.job_status_poller DEBUG: Adding executor multi
parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: <AppFuture at 0x15446395dd10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 40ae28ba7217b909e7ed07471576ff06
parsl.dataflow.memoization INFO: Task 0 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1
parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/09eebe30-3ada-4057-ba9e-2978d0fcfd43_isrForCrosstalkSources_855717_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/09eebe30-3ada-4057-ba9e-2978d0fcfd43_isrForCrosstalkSources_855717_27.stderr
parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: <AppFuture at 0x154463b167d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 649ad4f0da844d917cf4e521ade6a90e
parsl.dataflow.memoization INFO: Task 1 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2
parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/18f4ef95-1342-4234-8b2e-434e20665e5c_isrForCrosstalkSources_855716_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/18f4ef95-1342-4234-8b2e-434e20665e5c_isrForCrosstalkSources_855716_11.stderr
parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: <AppFuture at 0x15446e1acdd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 6f8cbd26c5e2e9dc3ad04e9f437990d8
parsl.dataflow.memoization INFO: Task 2 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3
parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/55d4de18-27a0-4dc6-9e6a-0425e4e561cb_isrForCrosstalkSources_855810_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/55d4de18-27a0-4dc6-9e6a-0425e4e561cb_isrForCrosstalkSources_855810_61.stderr
parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: <AppFuture at 0x154463ac6b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 9963a0312bdda7f9920651b42c2c5105
parsl.dataflow.memoization INFO: Task 3 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4
parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/95945658-1a14-4d93-9333-f84361dc4358_isrForCrosstalkSources_855717_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/95945658-1a14-4d93-9333-f84361dc4358_isrForCrosstalkSources_855717_1.stderr
parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: <AppFuture at 0x154463ac2110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 29444492f0761a46a8ffc5e0d8a81681
parsl.dataflow.memoization INFO: Task 4 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5
parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/8f0db560-94cd-458d-b4fa-d35fc1765b8f_isrForCrosstalkSources_855716_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/8f0db560-94cd-458d-b4fa-d35fc1765b8f_isrForCrosstalkSources_855716_14.stderr
parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: <AppFuture at 0x154463ac3050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 6ad2440a4d3bf37575e1dc63a6145474
parsl.dataflow.memoization INFO: Task 5 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6
parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/1de5d971-c596-4702-aed8-0725b51cc004_isrForCrosstalkSources_855810_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/1de5d971-c596-4702-aed8-0725b51cc004_isrForCrosstalkSources_855810_45.stderr
parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: <AppFuture at 0x154463ac3490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 5c2f77601f07b2bde362631adb35c751
parsl.dataflow.memoization INFO: Task 6 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7
parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/46c8b1d8-d4cd-48ba-be6b-017d6567523b_isrForCrosstalkSources_855810_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/46c8b1d8-d4cd-48ba-be6b-017d6567523b_isrForCrosstalkSources_855810_31.stderr
parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: <AppFuture at 0x154463aecd50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 887bd84f374e48aaf51145133aaeab31
parsl.dataflow.memoization INFO: Task 7 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8
parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/76e99c69-5cbc-457a-afe0-724c118c0430_isrForCrosstalkSources_855809_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/76e99c69-5cbc-457a-afe0-724c118c0430_isrForCrosstalkSources_855809_37.stderr
parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: <AppFuture at 0x154463a7e910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 8 has memoization hash ff14a604a31af0a605e4b2699aa4b09c
parsl.dataflow.memoization INFO: Task 8 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9
parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/02a48a2b-4731-4fb3-bb12-ab8371f80286_isrForCrosstalkSources_855717_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/02a48a2b-4731-4fb3-bb12-ab8371f80286_isrForCrosstalkSources_855717_20.stderr
parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: <AppFuture at 0x15446dee47d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 39eaa6b18f96a31dd413444c3f4ea180
parsl.dataflow.memoization INFO: Task 9 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10
parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/a5d27239-3931-42f1-8a31-75a33fbd1878_isrForCrosstalkSources_855716_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/a5d27239-3931-42f1-8a31-75a33fbd1878_isrForCrosstalkSources_855716_6.stderr
parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: <AppFuture at 0x154463a98550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 7df52f6be0d71b8465830020b8f832e4
parsl.dataflow.memoization INFO: Task 10 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11
parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/07da1648-b2e9-4dc0-bc89-a2fd2818fa3c_isrForCrosstalkSources_855717_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/07da1648-b2e9-4dc0-bc89-a2fd2818fa3c_isrForCrosstalkSources_855717_19.stderr
parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: <AppFuture at 0x154463af8fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 11 has memoization hash ae9e96844da8e2090e597cf6e618f274
parsl.dataflow.memoization INFO: Task 11 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12
parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/7cf151a2-0853-4760-91ba-1df3eec2cd23_isrForCrosstalkSources_855716_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/7cf151a2-0853-4760-91ba-1df3eec2cd23_isrForCrosstalkSources_855716_3.stderr
parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: <AppFuture at 0x154463a70450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 07a5be2cc7f75a522ccfc29ee70091c8
parsl.dataflow.memoization INFO: Task 12 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13
parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/ea4e9acc-7aa4-49ab-836c-c52c68eb5ab1_isrForCrosstalkSources_855809_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/ea4e9acc-7aa4-49ab-836c-c52c68eb5ab1_isrForCrosstalkSources_855809_48.stderr
parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: <AppFuture at 0x154463a70950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 5387da3f1bd2333a3530e67d17978494
parsl.dataflow.memoization INFO: Task 13 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14
parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/222efef2-61a9-4a2e-896e-d33a3d15ded1_isrForCrosstalkSources_855810_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/222efef2-61a9-4a2e-896e-d33a3d15ded1_isrForCrosstalkSources_855810_58.stderr
parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: <AppFuture at 0x154463aaa1d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 14 has memoization hash ed75f6c62c179e294aa51513b0aad489
parsl.dataflow.memoization INFO: Task 14 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15
parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/e4feae2e-0122-414c-b93a-8e78f52055af_isrForCrosstalkSources_855716_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/e4feae2e-0122-414c-b93a-8e78f52055af_isrForCrosstalkSources_855716_35.stderr
parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: <AppFuture at 0x154463aa9c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 0a96904054bc795a38b1a2fc324756d7
parsl.dataflow.memoization INFO: Task 15 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16
parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/0dab09a1-9088-43ae-b0b9-34f25f99bad7_isrForCrosstalkSources_855716_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/0dab09a1-9088-43ae-b0b9-34f25f99bad7_isrForCrosstalkSources_855716_4.stderr
parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: <AppFuture at 0x154463a8d250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 16 has memoization hash db99705cbc30e01f6874b9d1d62da340
parsl.dataflow.memoization INFO: Task 16 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17
parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/aeb2a75a-15f1-4b43-8fa7-9329f3e38608_isrForCrosstalkSources_855810_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/aeb2a75a-15f1-4b43-8fa7-9329f3e38608_isrForCrosstalkSources_855810_39.stderr
parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: <AppFuture at 0x154463a8f190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 1c61d1d713669bcfd68a4a68c4c9ff79
parsl.dataflow.memoization INFO: Task 17 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18
parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/aca64fdd-b67d-45c8-8714-b49beac981e2_isrForCrosstalkSources_855717_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/aca64fdd-b67d-45c8-8714-b49beac981e2_isrForCrosstalkSources_855717_33.stderr
parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: <AppFuture at 0x15446e17d5d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 6ff914cf2737624055d9f09c365b425a
parsl.dataflow.memoization INFO: Task 18 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19
parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/9f267cbb-16bb-4901-9127-bd1b03be102c_isrForCrosstalkSources_855717_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/9f267cbb-16bb-4901-9127-bd1b03be102c_isrForCrosstalkSources_855717_6.stderr
parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: <AppFuture at 0x154463ab4690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 9af76c7e0bbd6c3cf6c89a4a2333d9b5
parsl.dataflow.memoization INFO: Task 19 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20
parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/647245f5-1866-47ec-882c-bce615df20ed_isrForCrosstalkSources_855810_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/647245f5-1866-47ec-882c-bce615df20ed_isrForCrosstalkSources_855810_60.stderr
parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: <AppFuture at 0x154463a63490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 2648ddb424a02a5b7d1ff8011f01ce35
parsl.dataflow.memoization INFO: Task 20 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21
parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/79681c56-f258-44ef-b592-54d49d2040d8_isrForCrosstalkSources_855808_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/79681c56-f258-44ef-b592-54d49d2040d8_isrForCrosstalkSources_855808_10.stderr
parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: <AppFuture at 0x154463a62fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 0c76cd2c1b138e40525d27067ed23393
parsl.dataflow.memoization INFO: Task 21 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22
parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/52b5d730-60bb-4866-bb66-e8022b3e852c_isrForCrosstalkSources_855809_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/52b5d730-60bb-4866-bb66-e8022b3e852c_isrForCrosstalkSources_855809_41.stderr
parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: <AppFuture at 0x154463a63250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 22 has memoization hash cd524a4cc41d1cf62741741348ac8c08
parsl.dataflow.memoization INFO: Task 22 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23
parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/5a509f48-01da-4cb3-b3b1-604a4e81eac5_isrForCrosstalkSources_855718_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/5a509f48-01da-4cb3-b3b1-604a4e81eac5_isrForCrosstalkSources_855718_27.stderr
parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: <AppFuture at 0x154463a56c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 47ae7f6094524d763e7c76cce6096fc8
parsl.dataflow.memoization INFO: Task 23 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24
parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/66cd32b9-9387-47f7-9820-f869273dc0a8_isrForCrosstalkSources_855718_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/66cd32b9-9387-47f7-9820-f869273dc0a8_isrForCrosstalkSources_855718_31.stderr
parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: <AppFuture at 0x154463a54510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 0dc7a78cad6be00a4e0e82330b5c8706
parsl.dataflow.memoization INFO: Task 24 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25
parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/8b47d805-40fb-46b2-bdd8-714e04896ac9_isrForCrosstalkSources_855716_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/8b47d805-40fb-46b2-bdd8-714e04896ac9_isrForCrosstalkSources_855716_38.stderr
parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: <AppFuture at 0x154463a4ec10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 25 has memoization hash ecca87c9feaf994d2d85d388aaf3e6c8
parsl.dataflow.memoization INFO: Task 25 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26
parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/3810d204-3b2b-4cfb-aa6b-4525dc99b89a_isrForCrosstalkSources_855809_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/3810d204-3b2b-4cfb-aa6b-4525dc99b89a_isrForCrosstalkSources_855809_20.stderr
parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: <AppFuture at 0x154463a4ce10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 26 has memoization hash a0cf8a58a200a34925ffcc83e5aef8f8
parsl.dataflow.memoization INFO: Task 26 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27
parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/0a01f56b-7905-414b-9dc6-5d7207f69e28_isrForCrosstalkSources_855809_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/0a01f56b-7905-414b-9dc6-5d7207f69e28_isrForCrosstalkSources_855809_26.stderr
parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: <AppFuture at 0x154463a4d450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 27 has memoization hash ab11ff9433576da15c633cef45eca83b
parsl.dataflow.memoization INFO: Task 27 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28
parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/0001b779-175b-4b96-af36-76e7993493e3_isrForCrosstalkSources_855810_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/0001b779-175b-4b96-af36-76e7993493e3_isrForCrosstalkSources_855810_5.stderr
parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: <AppFuture at 0x154463a3cbd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 2945061cc4a8dbcc786b6b1f8c6ba4fe
parsl.dataflow.memoization INFO: Task 28 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29
parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/60af3367-db8e-42a6-9fbd-e848665b8351_isrForCrosstalkSources_855809_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/60af3367-db8e-42a6-9fbd-e848665b8351_isrForCrosstalkSources_855809_34.stderr
parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: <AppFuture at 0x154463a3e610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 29 has memoization hash c3ca2a0e11aba690bb4b79d3836f9df4
parsl.dataflow.memoization INFO: Task 29 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30
parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/c6bf93ab-79b1-4c90-a770-562cc0dd03ea_isrForCrosstalkSources_855716_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/c6bf93ab-79b1-4c90-a770-562cc0dd03ea_isrForCrosstalkSources_855716_46.stderr
parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: <AppFuture at 0x154463a3eb50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 48dbac8e9a1aa95e5aedd41d5e908724
parsl.dataflow.memoization INFO: Task 30 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31
parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/d2a44e05-f80c-4052-b0bb-38a597318dd8_isrForCrosstalkSources_855808_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/d2a44e05-f80c-4052-b0bb-38a597318dd8_isrForCrosstalkSources_855808_46.stderr
parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: <AppFuture at 0x154463a3b5d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 07622cdfb2e4af7e39f118d1a078d75d
parsl.dataflow.memoization INFO: Task 31 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32
parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/76b76337-a41e-47e1-971b-8e67509d617d_isrForCrosstalkSources_855808_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/76b76337-a41e-47e1-971b-8e67509d617d_isrForCrosstalkSources_855808_41.stderr
parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: <AppFuture at 0x154463a39090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 32 has memoization hash ee91e50fac6ecab3366c95abd1df8da5
parsl.dataflow.memoization INFO: Task 32 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33
parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/5e7914de-62d4-483d-95d3-2689dbab96aa_isrForCrosstalkSources_855809_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/5e7914de-62d4-483d-95d3-2689dbab96aa_isrForCrosstalkSources_855809_47.stderr
parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: <AppFuture at 0x154463a3b6d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 9a319890af47cd32a1ce510184f0a01a
parsl.dataflow.memoization INFO: Task 33 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34
parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/e5d316b0-ccb5-4f8e-9fa4-7bbfebca3c0b_isrForCrosstalkSources_855718_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/e5d316b0-ccb5-4f8e-9fa4-7bbfebca3c0b_isrForCrosstalkSources_855718_26.stderr
parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: <AppFuture at 0x154463a2c450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 6903e1968552ef172382d67b0c92dfa0
parsl.dataflow.memoization INFO: Task 34 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35
parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/c8f50f66-92ba-441d-849f-717b50ee6f9c_isrForCrosstalkSources_855810_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/c8f50f66-92ba-441d-849f-717b50ee6f9c_isrForCrosstalkSources_855810_24.stderr
parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: <AppFuture at 0x154463a2fad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 35 has memoization hash d36febe6616ff2ad90197bf2e08e79a7
parsl.dataflow.memoization INFO: Task 35 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36
parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/0dca4b0a-abac-4a96-ad25-641fe04e4418_isrForCrosstalkSources_855809_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/0dca4b0a-abac-4a96-ad25-641fe04e4418_isrForCrosstalkSources_855809_52.stderr
parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: <AppFuture at 0x154463a2e050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 36 has memoization hash a57146dd734ac999f4b8ed02bb398a57
parsl.dataflow.memoization INFO: Task 36 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37
parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/794c6658-c57a-450d-a8ce-07caad22a049_isrForCrosstalkSources_855717_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/794c6658-c57a-450d-a8ce-07caad22a049_isrForCrosstalkSources_855717_44.stderr
parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: <AppFuture at 0x154463a8c1d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 15ebb43bfd00563b6d95baa69bf07f18
parsl.dataflow.memoization INFO: Task 37 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38
parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/d0fdc23a-b571-4fd0-8437-7035ee877a26_isrForCrosstalkSources_855717_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/d0fdc23a-b571-4fd0-8437-7035ee877a26_isrForCrosstalkSources_855717_54.stderr
parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: <AppFuture at 0x154463a1db50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 38 has memoization hash e72f617fac7d751c98318235f62f34ad
parsl.dataflow.memoization INFO: Task 38 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39
parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/ee41f1a9-30f4-400b-910c-26af5e7cf048_isrForCrosstalkSources_855718_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/ee41f1a9-30f4-400b-910c-26af5e7cf048_isrForCrosstalkSources_855718_61.stderr
parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: <AppFuture at 0x154463a1d050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 39 has memoization hash e210935d0d7cfe11e66a70b7b98880ce
parsl.dataflow.memoization INFO: Task 39 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40
parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/4612ccef-0d84-4a2a-a549-a89de00e9f7a_isrForCrosstalkSources_855809_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/4612ccef-0d84-4a2a-a549-a89de00e9f7a_isrForCrosstalkSources_855809_16.stderr
parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: <AppFuture at 0x154463ac6990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 40 has memoization hash f0a223ea268eeadf0a0011bb27e1e5a5
parsl.dataflow.memoization INFO: Task 40 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41
parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/dbd57a2f-2a07-4f08-9815-0ceb97d5006b_isrForCrosstalkSources_855718_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/dbd57a2f-2a07-4f08-9815-0ceb97d5006b_isrForCrosstalkSources_855718_30.stderr
parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: <AppFuture at 0x154463c14e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 166dc5c2d38a4231010edaf7c38ebb70
parsl.dataflow.memoization INFO: Task 41 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42
parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/166b47c9-fe35-4bef-9bdd-af5f4b0e8166_isrForCrosstalkSources_855809_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/166b47c9-fe35-4bef-9bdd-af5f4b0e8166_isrForCrosstalkSources_855809_3.stderr
parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: <AppFuture at 0x154463c17e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 42 has memoization hash fabf2d4a28b381a8e78b756d240e75e2
parsl.dataflow.memoization INFO: Task 42 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43
parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/776cb1ff-f14d-43bb-9f2d-be0415658aee_isrForCrosstalkSources_855717_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/776cb1ff-f14d-43bb-9f2d-be0415658aee_isrForCrosstalkSources_855717_4.stderr
parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: <AppFuture at 0x154463c14690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 22a41d9d59cbb083db7ba9fdf15fe2ce
parsl.dataflow.memoization INFO: Task 43 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44
parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/0dbbbf0e-1146-40ea-a750-9b84d9b4b996_isrForCrosstalkSources_855717_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/0dbbbf0e-1146-40ea-a750-9b84d9b4b996_isrForCrosstalkSources_855717_46.stderr
parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: <AppFuture at 0x154463c04b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 061aea2e94e90263965b5b0dfad6c711
parsl.dataflow.memoization INFO: Task 44 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45
parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/d2c77852-7ae0-4ff4-aa00-1e8c09b78114_isrForCrosstalkSources_855809_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/d2c77852-7ae0-4ff4-aa00-1e8c09b78114_isrForCrosstalkSources_855809_4.stderr
parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: <AppFuture at 0x154463c05850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 829f7dc4c284d3f9dd62e01c24a10c25
parsl.dataflow.memoization INFO: Task 45 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46
parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/c3300561-3486-4e63-83c3-adff3488dda4_isrForCrosstalkSources_855718_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/c3300561-3486-4e63-83c3-adff3488dda4_isrForCrosstalkSources_855718_2.stderr
parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: <AppFuture at 0x154463c06c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 46 has memoization hash b6668669183c4f12d9238a34dcb4f461
parsl.dataflow.memoization INFO: Task 46 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47
parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/43e9dc8b-c23e-4e90-a53a-d6e8256e4c60_isrForCrosstalkSources_855716_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/43e9dc8b-c23e-4e90-a53a-d6e8256e4c60_isrForCrosstalkSources_855716_8.stderr
parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: <AppFuture at 0x154463c06290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 6adc5e0f79dc2b3097328d2040a759f5
parsl.dataflow.memoization INFO: Task 47 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48
parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/f6cb9742-6ea7-4cfa-97fa-42e84ef748e6_isrForCrosstalkSources_855718_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/f6cb9742-6ea7-4cfa-97fa-42e84ef748e6_isrForCrosstalkSources_855718_38.stderr
parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: <AppFuture at 0x154463bfca50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 395c6022941d26558daed5b819bb59a6
parsl.dataflow.memoization INFO: Task 48 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49
parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/cfb36cab-963d-4c52-8b12-11a689fbc642_isrForCrosstalkSources_855810_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/cfb36cab-963d-4c52-8b12-11a689fbc642_isrForCrosstalkSources_855810_53.stderr
parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: <AppFuture at 0x154463bff810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 49 has memoization hash bddd505cabb1a2c3370c2e12a37cbeaf
parsl.dataflow.memoization INFO: Task 49 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50
parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/aa2bd463-71db-4003-b494-ca120733e8c5_isrForCrosstalkSources_855716_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/aa2bd463-71db-4003-b494-ca120733e8c5_isrForCrosstalkSources_855716_16.stderr
parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: <AppFuture at 0x154463bf2dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 4686aeb64f6933a3c38afd6ef3566550
parsl.dataflow.memoization INFO: Task 50 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51
parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/198be7e5-42c4-4110-a0c5-217b67e74085_isrForCrosstalkSources_855717_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/198be7e5-42c4-4110-a0c5-217b67e74085_isrForCrosstalkSources_855717_42.stderr
parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: <AppFuture at 0x154463bf0f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 87866e70a763c5a83f3a8a01a1e87a07
parsl.dataflow.memoization INFO: Task 51 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52
parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/0653461c-b488-4769-b191-4382b58b75c6_isrForCrosstalkSources_855810_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/0653461c-b488-4769-b191-4382b58b75c6_isrForCrosstalkSources_855810_29.stderr
parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: <AppFuture at 0x154463bf25d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 52 has memoization hash eefa566046de88e8da78268d38582f00
parsl.dataflow.memoization INFO: Task 52 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53
parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/e23fb8ef-2d6c-41ff-99ab-d0d8d51480e7_isrForCrosstalkSources_855718_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/e23fb8ef-2d6c-41ff-99ab-d0d8d51480e7_isrForCrosstalkSources_855718_46.stderr
parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: <AppFuture at 0x154463bd8710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 53 has memoization hash beb5e700bc2b38cd01184fc4e77c3268
parsl.dataflow.memoization INFO: Task 53 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54
parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/ce5c93c4-c7af-41a3-956a-bebd58fb13ea_isrForCrosstalkSources_855810_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/ce5c93c4-c7af-41a3-956a-bebd58fb13ea_isrForCrosstalkSources_855810_8.stderr
parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: <AppFuture at 0x154463bd8dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 54 has memoization hash abffc2addffe1991fdf17b6b104060d7
parsl.dataflow.memoization INFO: Task 54 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55
parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/fd2abb0d-82ff-4fa1-b916-e17c1e495bfb_isrForCrosstalkSources_855716_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/fd2abb0d-82ff-4fa1-b916-e17c1e495bfb_isrForCrosstalkSources_855716_54.stderr
parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: <AppFuture at 0x154463bdad50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 55 has memoization hash a4c3959c8f2321e221ff6d854f288cc7
parsl.dataflow.memoization INFO: Task 55 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56
parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/cea155c1-dee8-4d9f-bbd5-7a1f5de92c1d_isrForCrosstalkSources_855716_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/cea155c1-dee8-4d9f-bbd5-7a1f5de92c1d_isrForCrosstalkSources_855716_29.stderr
parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: <AppFuture at 0x154463bd7f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 56 has memoization hash c949a471d980d2f1bb5c3f6b59c74fe9
parsl.dataflow.memoization INFO: Task 56 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57
parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/f1a11b4d-3f3c-413a-80b6-d8211f1d2740_isrForCrosstalkSources_855808_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/f1a11b4d-3f3c-413a-80b6-d8211f1d2740_isrForCrosstalkSources_855808_6.stderr
parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: <AppFuture at 0x154463ab5590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 57 has memoization hash b225a6a3c48300af1da1dcf4b6a603c8
parsl.dataflow.memoization INFO: Task 57 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58
parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/da55b8ca-cf85-4480-a561-784dbf1a8d1a_isrForCrosstalkSources_855810_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/da55b8ca-cf85-4480-a561-784dbf1a8d1a_isrForCrosstalkSources_855810_40.stderr
parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: <AppFuture at 0x154463bd52d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 0595f31bf42121c15a9b7babaee5b55b
parsl.dataflow.memoization INFO: Task 58 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59
parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/caac27a7-2331-4ed5-879a-262565331a77_isrForCrosstalkSources_855716_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/caac27a7-2331-4ed5-879a-262565331a77_isrForCrosstalkSources_855716_5.stderr
parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: <AppFuture at 0x154463a981d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 6fea6aeb886e5b362290ab315eafbea5
parsl.dataflow.memoization INFO: Task 59 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637a71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60
parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/01276b26-4bab-4fa2-a763-adf3a0a7c442_isrForCrosstalkSources_855718_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/01276b26-4bab-4fa2-a763-adf3a0a7c442_isrForCrosstalkSources_855718_56.stderr
parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: <AppFuture at 0x154463bc9850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 60 has memoization hash c432e4052839e641bbef0a17c52860ab
parsl.dataflow.memoization INFO: Task 60 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61
parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/e4d61c22-0f03-4351-a633-3b3f78fec695_isrForCrosstalkSources_855716_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/e4d61c22-0f03-4351-a633-3b3f78fec695_isrForCrosstalkSources_855716_30.stderr
parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: <AppFuture at 0x154463bc8f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 1040372fd60f0e7185038a89b2a92773
parsl.dataflow.memoization INFO: Task 61 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62
parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/f7ebd5fe-319e-4434-8471-e2672aeeef86_isrForCrosstalkSources_855717_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/f7ebd5fe-319e-4434-8471-e2672aeeef86_isrForCrosstalkSources_855717_26.stderr
parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: <AppFuture at 0x154463bc8190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 62 has memoization hash c5c02a1b27b918cab49399c1079bc8c8
parsl.dataflow.memoization INFO: Task 62 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63
parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/2335573c-64d5-41db-acb3-78adf94d9a87_isrForCrosstalkSources_855810_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/2335573c-64d5-41db-acb3-78adf94d9a87_isrForCrosstalkSources_855810_22.stderr
parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: <AppFuture at 0x154463bbded0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 2312537624ef7e47924fce07ad70f769
parsl.dataflow.memoization INFO: Task 63 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64
parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/435f723a-37a2-4c7f-976e-369415eb756d_isrForCrosstalkSources_855717_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/435f723a-37a2-4c7f-976e-369415eb756d_isrForCrosstalkSources_855717_13.stderr
parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: <AppFuture at 0x154463bbe010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 566dc2d10c03d43c28761e537a52c66f
parsl.dataflow.memoization INFO: Task 64 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65
parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/7de08bde-143e-43ed-91d6-d42f77e2b1f2_isrForCrosstalkSources_855716_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/7de08bde-143e-43ed-91d6-d42f77e2b1f2_isrForCrosstalkSources_855716_57.stderr
parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: <AppFuture at 0x154463bbcb10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 5e5e33e08459aeff49ae89d33dd83e76
parsl.dataflow.memoization INFO: Task 65 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66
parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/ad81d948-66c1-49cf-9a15-45e3605b72d9_isrForCrosstalkSources_855718_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/ad81d948-66c1-49cf-9a15-45e3605b72d9_isrForCrosstalkSources_855718_45.stderr
parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: <AppFuture at 0x154463bb5d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 6cf88871ee6df787f3257bbdb8472bdf
parsl.dataflow.memoization INFO: Task 66 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67
parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/09729c3b-674c-4d95-b781-7b7788f18cdf_isrForCrosstalkSources_855808_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/09729c3b-674c-4d95-b781-7b7788f18cdf_isrForCrosstalkSources_855808_9.stderr
parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: <AppFuture at 0x154463bb7b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 5c307fe04b4e44b64db678d6110346ea
parsl.dataflow.memoization INFO: Task 67 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68
parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/2deda282-7e75-42f6-be0b-cc23347e91d4_isrForCrosstalkSources_855717_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/2deda282-7e75-42f6-be0b-cc23347e91d4_isrForCrosstalkSources_855717_25.stderr
parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: <AppFuture at 0x154463bb6710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 059102b5ecff080400c40d44d538fcb6
parsl.dataflow.memoization INFO: Task 68 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69
parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/ad209952-fb69-4ef3-a60b-fe9fcc38e1ce_isrForCrosstalkSources_855718_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/ad209952-fb69-4ef3-a60b-fe9fcc38e1ce_isrForCrosstalkSources_855718_50.stderr
parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: <AppFuture at 0x154463bb5e90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 69 has memoization hash db743d22b6dc1a0d345009ca0e6c52d7
parsl.dataflow.memoization INFO: Task 69 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70
parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/6aa09c6b-cc69-4f35-a4ba-0b1ff713ffd8_isrForCrosstalkSources_855717_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/6aa09c6b-cc69-4f35-a4ba-0b1ff713ffd8_isrForCrosstalkSources_855717_15.stderr
parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: <AppFuture at 0x154463ba7310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 70 has memoization hash df7eeb5615ccf5d3ef79b273c30eb666
parsl.dataflow.memoization INFO: Task 70 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71
parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/6d228380-6472-446d-a86f-93a11a023617_isrForCrosstalkSources_855717_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/6d228380-6472-446d-a86f-93a11a023617_isrForCrosstalkSources_855717_37.stderr
parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: <AppFuture at 0x154463ba4bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 10b3efcc3935dddabe7e14b6f8469e2a
parsl.dataflow.memoization INFO: Task 71 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72
parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/40e56478-b601-485c-8b66-150bd5b4f6b0_isrForCrosstalkSources_855716_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/40e56478-b601-485c-8b66-150bd5b4f6b0_isrForCrosstalkSources_855716_55.stderr
parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: <AppFuture at 0x154463a1cb90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 72 has memoization hash fc8d6737668eecfcffe1c0c04da3b766
parsl.dataflow.memoization INFO: Task 72 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73
parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/5a5bb503-e84a-416c-bb4b-90f074ba3e2b_isrForCrosstalkSources_855808_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/5a5bb503-e84a-416c-bb4b-90f074ba3e2b_isrForCrosstalkSources_855808_32.stderr
parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: <AppFuture at 0x154463b9d150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 73 has memoization hash e36a2ebbe76df142af87f29ee2c33432
parsl.dataflow.memoization INFO: Task 73 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74
parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/76111870-61cf-4237-a4d1-0d03ae25e7d3_isrForCrosstalkSources_855717_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/76111870-61cf-4237-a4d1-0d03ae25e7d3_isrForCrosstalkSources_855717_18.stderr
parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: <AppFuture at 0x154463b9cdd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 22f009f292402ff49862d263ca419c3a
parsl.dataflow.memoization INFO: Task 74 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75
parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/de91162e-9dd1-4a6b-92c9-6897e1ca0e23_isrForCrosstalkSources_855717_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/de91162e-9dd1-4a6b-92c9-6897e1ca0e23_isrForCrosstalkSources_855717_28.stderr
parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: <AppFuture at 0x154463b9f750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 19c2c0ce1cd855119b5d0c8a00ea79f3
parsl.dataflow.memoization INFO: Task 75 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76
parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/25e524dd-049b-41ee-86f8-426d6defa23b_isrForCrosstalkSources_855808_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/25e524dd-049b-41ee-86f8-426d6defa23b_isrForCrosstalkSources_855808_15.stderr
parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: <AppFuture at 0x154463b91c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 515ccd8423ca647127d5aac0f71ac6ca
parsl.dataflow.memoization INFO: Task 76 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77
parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/ff17624b-6839-4b8c-bf35-3fb3e5f5e514_isrForCrosstalkSources_855717_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/ff17624b-6839-4b8c-bf35-3fb3e5f5e514_isrForCrosstalkSources_855717_57.stderr
parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: <AppFuture at 0x154463b90c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 77 has memoization hash 7eca6add3854dad056bcb6fa52f80404
parsl.dataflow.memoization INFO: Task 77 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78
parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/387cdca1-cfd5-4c0d-b8f0-c23fb320e13d_isrForCrosstalkSources_855808_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/387cdca1-cfd5-4c0d-b8f0-c23fb320e13d_isrForCrosstalkSources_855808_58.stderr
parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: <AppFuture at 0x154463b91f10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 78 has memoization hash d153b5d46e8fd6bf3c9b56b343a8b6fc
parsl.dataflow.memoization INFO: Task 78 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79
parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/c9a8ddd0-ff74-4efb-8762-e3aace0cc9b0_isrForCrosstalkSources_855717_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/c9a8ddd0-ff74-4efb-8762-e3aace0cc9b0_isrForCrosstalkSources_855717_32.stderr
parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: <AppFuture at 0x154463b92650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 79 has memoization hash ecf3dd37abeb8bfe7976e8c09e70cfa1
parsl.dataflow.memoization INFO: Task 79 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80
parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/639f7933-237e-47d5-bd0c-c94d0effcafb_isrForCrosstalkSources_855716_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/639f7933-237e-47d5-bd0c-c94d0effcafb_isrForCrosstalkSources_855716_53.stderr
parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: <AppFuture at 0x154463b87050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 80 has memoization hash a110ee4ce913948ffec2d3ef833ff42a
parsl.dataflow.memoization INFO: Task 80 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81
parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/2486b497-b695-4067-b1e8-d527d5b9667f_isrForCrosstalkSources_855718_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/2486b497-b695-4067-b1e8-d527d5b9667f_isrForCrosstalkSources_855718_17.stderr
parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: <AppFuture at 0x154463b86150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 81 has memoization hash c333885b6c6598fd219ee701fc0c48d5
parsl.dataflow.memoization INFO: Task 81 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82
parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/dc764e95-5a8d-4cb1-bfa0-6e076f4cb408_isrForCrosstalkSources_855716_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/dc764e95-5a8d-4cb1-bfa0-6e076f4cb408_isrForCrosstalkSources_855716_41.stderr
parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: <AppFuture at 0x154463b87c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 76af9f007f7a42da191cd25a2a19ff5f
parsl.dataflow.memoization INFO: Task 82 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83
parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/cea8637a-f1c3-40d6-bf0f-05f1e1d76ae9_isrForCrosstalkSources_855808_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/cea8637a-f1c3-40d6-bf0f-05f1e1d76ae9_isrForCrosstalkSources_855808_4.stderr
parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: <AppFuture at 0x154463b867d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 99c91126de6bd47352e3aa7c4ae4c4cb
parsl.dataflow.memoization INFO: Task 83 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84
parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/1942f747-7e79-4c62-b2a3-b6c704e306ba_isrForCrosstalkSources_855718_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/1942f747-7e79-4c62-b2a3-b6c704e306ba_isrForCrosstalkSources_855718_12.stderr
parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: <AppFuture at 0x154463bf1410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 84 has memoization hash eacc0aed9cf1e4591edf22539e49dd5f
parsl.dataflow.memoization INFO: Task 84 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85
parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/d1cae28d-2310-4699-88a5-2d4156e9ecd6_isrForCrosstalkSources_855809_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/d1cae28d-2310-4699-88a5-2d4156e9ecd6_isrForCrosstalkSources_855809_40.stderr
parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: <AppFuture at 0x154463b7ac10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 85 has memoization hash c3b7fbb532ce1f891dc6c847c36d61e7
parsl.dataflow.memoization INFO: Task 85 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86
parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/838e2cbf-e2ff-4aa1-8d7f-d5c244e07cb0_isrForCrosstalkSources_855810_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/838e2cbf-e2ff-4aa1-8d7f-d5c244e07cb0_isrForCrosstalkSources_855810_50.stderr
parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: <AppFuture at 0x154463b79850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 9d1d2e8e5aa8afc6aa0c73fe1f9f30b4
parsl.dataflow.memoization INFO: Task 86 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87
parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/1c325253-24bb-41c2-a794-b054a78cfbc6_isrForCrosstalkSources_855810_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/1c325253-24bb-41c2-a794-b054a78cfbc6_isrForCrosstalkSources_855810_48.stderr
parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: <AppFuture at 0x154463b7b050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 6556ea6835908cca748df916dc2e4cde
parsl.dataflow.memoization INFO: Task 87 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88
parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/4726fb06-90fe-4467-a64b-f50672469e9f_isrForCrosstalkSources_855808_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/4726fb06-90fe-4467-a64b-f50672469e9f_isrForCrosstalkSources_855808_8.stderr
parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: <AppFuture at 0x154463b78d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 55bcf55d96fa48e95dce6e72717f152c
parsl.dataflow.memoization INFO: Task 88 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89
parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/de2e2f80-52ae-4f9d-a781-db3f6e9a3cc9_isrForCrosstalkSources_855808_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/de2e2f80-52ae-4f9d-a781-db3f6e9a3cc9_isrForCrosstalkSources_855808_39.stderr
parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: <AppFuture at 0x154463b71910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 4f8d1c12f3868c9a7c1c58e72ed8b464
parsl.dataflow.memoization INFO: Task 89 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90
parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/c5f3148e-6736-4936-a5eb-b63255155017_isrForCrosstalkSources_855717_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/c5f3148e-6736-4936-a5eb-b63255155017_isrForCrosstalkSources_855717_48.stderr
parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: <AppFuture at 0x154463b71950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 0b7ea1e6a09108a254bf821764b6c85a
parsl.dataflow.memoization INFO: Task 90 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91
parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/932a6caa-514d-4a83-b278-db8c97f8edbf_isrForCrosstalkSources_855809_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/932a6caa-514d-4a83-b278-db8c97f8edbf_isrForCrosstalkSources_855809_22.stderr
parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: <AppFuture at 0x154463b70a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 6fb237f908ac5b45985cca2c0847e8d1
parsl.dataflow.memoization INFO: Task 91 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92
parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/77405b0e-dfd2-4103-97b0-b72c084879d4_isrForCrosstalkSources_855718_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/77405b0e-dfd2-4103-97b0-b72c084879d4_isrForCrosstalkSources_855718_59.stderr
parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: <AppFuture at 0x154463bf3210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 92 has memoization hash a405d1cf95b7967bc19dbf33576cbfea
parsl.dataflow.memoization INFO: Task 92 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93
parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/890086b7-433b-4b05-833f-1c5175fb41c2_isrForCrosstalkSources_855808_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/890086b7-433b-4b05-833f-1c5175fb41c2_isrForCrosstalkSources_855808_18.stderr
parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: <AppFuture at 0x154463b68350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 93 has memoization hash a1055bfa398f7b2af358c8d8c61c701d
parsl.dataflow.memoization INFO: Task 93 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94
parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/3801c6fe-feb9-4603-95ce-73e7fe4801f0_isrForCrosstalkSources_855718_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/3801c6fe-feb9-4603-95ce-73e7fe4801f0_isrForCrosstalkSources_855718_8.stderr
parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: <AppFuture at 0x154463b6ae90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 1a27fb23ff7426ba0b45303a59488391
parsl.dataflow.memoization INFO: Task 94 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95
parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/ead72d24-2bb3-4083-bcd0-32b527ad65de_isrForCrosstalkSources_855809_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/ead72d24-2bb3-4083-bcd0-32b527ad65de_isrForCrosstalkSources_855809_23.stderr
parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: <AppFuture at 0x154463b69610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 63d0bc9c2f956772a39a2cf1ae61eae1
parsl.dataflow.memoization INFO: Task 95 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96
parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/4773f8dd-5944-4eb9-b0e7-d1e9997b945a_isrForCrosstalkSources_855718_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/4773f8dd-5944-4eb9-b0e7-d1e9997b945a_isrForCrosstalkSources_855718_37.stderr
parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: <AppFuture at 0x154463b61d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 96 has memoization hash d43ea30d8dacfc20a3f344efff24fd4f
parsl.dataflow.memoization INFO: Task 96 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97
parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/27255655-219f-47e0-93b5-b7b6f12c132e_isrForCrosstalkSources_855808_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/27255655-219f-47e0-93b5-b7b6f12c132e_isrForCrosstalkSources_855808_56.stderr
parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: <AppFuture at 0x154463b63a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 2eb7682bae08e258061d3466ada410d2
parsl.dataflow.memoization INFO: Task 97 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98
parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/0570181c-3633-4233-8f93-9b9e2aeef8a0_isrForCrosstalkSources_855808_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/0570181c-3633-4233-8f93-9b9e2aeef8a0_isrForCrosstalkSources_855808_27.stderr
parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: <AppFuture at 0x154463b61050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 98 has memoization hash fbde43215834c77d57920ca21f4b2628
parsl.dataflow.memoization INFO: Task 98 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99
parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/29d88c3a-8c57-43b4-8672-d4dd115b0492_isrForCrosstalkSources_855810_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/29d88c3a-8c57-43b4-8672-d4dd115b0492_isrForCrosstalkSources_855810_3.stderr
parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: <AppFuture at 0x154463b68bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 704e5e0a6741a3d9142a56bb21ef3d22
parsl.dataflow.memoization INFO: Task 99 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100
parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/a8753765-698c-4842-825f-4c7c6f9b4004_isrForCrosstalkSources_855810_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/a8753765-698c-4842-825f-4c7c6f9b4004_isrForCrosstalkSources_855810_37.stderr
parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: <AppFuture at 0x154463df3d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 9f78e5d45700ed4e1160b02ffe734a5e
parsl.dataflow.memoization INFO: Task 100 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101
parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/5668ae2e-9224-4832-b0bc-b4f966ce6098_isrForCrosstalkSources_855716_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/5668ae2e-9224-4832-b0bc-b4f966ce6098_isrForCrosstalkSources_855716_24.stderr
parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: <AppFuture at 0x154463a60590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 9fb2da88c459a3e2beba4e08fb25dcdb
parsl.dataflow.memoization INFO: Task 101 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102
parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/ab1f7733-867c-4760-9685-58fdb207b728_isrForCrosstalkSources_855718_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/ab1f7733-867c-4760-9685-58fdb207b728_isrForCrosstalkSources_855718_36.stderr
parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: <AppFuture at 0x154463b5b210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 6dfffc882bfbd622afcdaeaea02700e8
parsl.dataflow.memoization INFO: Task 102 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103
parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/5cffef1b-1cdc-446b-b92d-53bce7c7bee4_isrForCrosstalkSources_855717_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/5cffef1b-1cdc-446b-b92d-53bce7c7bee4_isrForCrosstalkSources_855717_58.stderr
parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: <AppFuture at 0x154463b5a290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 52b9104bd72ef23b39e9bacb272a8374
parsl.dataflow.memoization INFO: Task 103 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104
parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/e35cde94-b5f0-42e0-a32e-9c8b9366cb21_isrForCrosstalkSources_855809_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/e35cde94-b5f0-42e0-a32e-9c8b9366cb21_isrForCrosstalkSources_855809_1.stderr
parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: <AppFuture at 0x154463b5ab10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 5779d685f0cbe0df5229cf81fc9d652d
parsl.dataflow.memoization INFO: Task 104 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105
parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/861c5309-07c7-4aff-82ec-8bbeec69a243_isrForCrosstalkSources_855716_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/861c5309-07c7-4aff-82ec-8bbeec69a243_isrForCrosstalkSources_855716_43.stderr
parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: <AppFuture at 0x154463b26990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 24d57b6d822f1076eba2d764c994be60
parsl.dataflow.memoization INFO: Task 105 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106
parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/42f118c9-802d-4d16-9eba-36cb926139a4_isrForCrosstalkSources_855809_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/42f118c9-802d-4d16-9eba-36cb926139a4_isrForCrosstalkSources_855809_61.stderr
parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: <AppFuture at 0x154463bd7090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 106 has memoization hash b880ef489cb61ac3eb7672587aa110ab
parsl.dataflow.memoization INFO: Task 106 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107
parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/e5d03eb9-e4c2-4d1f-9a71-1f5f6173d13c_isrForCrosstalkSources_855810_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/e5d03eb9-e4c2-4d1f-9a71-1f5f6173d13c_isrForCrosstalkSources_855810_17.stderr
parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: <AppFuture at 0x154463b3ff90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 107 has memoization hash a68d45ff6f18de167d564642ed595244
parsl.dataflow.memoization INFO: Task 107 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108
parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/bc96b4f1-780e-4900-a398-b16320dc5a40_isrForCrosstalkSources_855808_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/bc96b4f1-780e-4900-a398-b16320dc5a40_isrForCrosstalkSources_855808_38.stderr
parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: <AppFuture at 0x154463b3e5d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 0377336f799a7856f049e0974a649a73
parsl.dataflow.memoization INFO: Task 108 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109
parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/fbaaf73e-514b-42da-8c37-e5eda7bf5362_isrForCrosstalkSources_855810_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/fbaaf73e-514b-42da-8c37-e5eda7bf5362_isrForCrosstalkSources_855810_16.stderr
parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: <AppFuture at 0x154463b3c4d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 109 has memoization hash def58f29d7883db8bfa3d8a72f72b95b
parsl.dataflow.memoization INFO: Task 109 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110
parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/4032a593-4129-4935-9e8e-75733bba80be_isrForCrosstalkSources_855718_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/4032a593-4129-4935-9e8e-75733bba80be_isrForCrosstalkSources_855718_13.stderr
parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: <AppFuture at 0x154463b3d390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 23631e0a3cf4bd57a125871f9eaacc88
parsl.dataflow.memoization INFO: Task 110 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111
parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/4c33492c-830e-47c5-a0cd-09ca77def43b_isrForCrosstalkSources_855717_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/4c33492c-830e-47c5-a0cd-09ca77def43b_isrForCrosstalkSources_855717_22.stderr
parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: <AppFuture at 0x154463b3ead0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 111 has memoization hash e142502268f3e07b8d794f3605aa8502
parsl.dataflow.memoization INFO: Task 111 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112
parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/68148ff7-0abe-4fb6-91c8-b49fd04d54b3_isrForCrosstalkSources_855809_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/68148ff7-0abe-4fb6-91c8-b49fd04d54b3_isrForCrosstalkSources_855809_42.stderr
parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: <AppFuture at 0x154463b567d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 4fbafee0534b0590114603130d4c7ee9
parsl.dataflow.memoization INFO: Task 112 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113
parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/439c89c7-ef3c-4324-857c-3e98a39e4959_isrForCrosstalkSources_855809_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/439c89c7-ef3c-4324-857c-3e98a39e4959_isrForCrosstalkSources_855809_33.stderr
parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: <AppFuture at 0x154463b56210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 7b4506d1d95605c2d4ef478f49b72a05
parsl.dataflow.memoization INFO: Task 113 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114
parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/7e6304e4-5cf2-4b06-906e-590dc5345a54_isrForCrosstalkSources_855718_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/7e6304e4-5cf2-4b06-906e-590dc5345a54_isrForCrosstalkSources_855718_32.stderr
parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: <AppFuture at 0x154463b54490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 4a56164af847d9f6267133510679acbc
parsl.dataflow.memoization INFO: Task 114 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115
parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/ba839ae2-f7c1-40f9-9ce5-9da18637aab9_isrForCrosstalkSources_855810_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/ba839ae2-f7c1-40f9-9ce5-9da18637aab9_isrForCrosstalkSources_855810_54.stderr
parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: <AppFuture at 0x154463b4a690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 1f032b02cfd366f00e8711a2b98545ce
parsl.dataflow.memoization INFO: Task 115 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116
parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/0a172a13-837a-477a-ade1-e4b220c696e1_isrForCrosstalkSources_855808_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/0a172a13-837a-477a-ade1-e4b220c696e1_isrForCrosstalkSources_855808_61.stderr
parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: <AppFuture at 0x154463b43fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 34a58b4a4e6e930627125c996079c0ec
parsl.dataflow.memoization INFO: Task 116 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117
parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/19577493-09e9-42e3-a2a5-fbdf0c57cb96_isrForCrosstalkSources_855808_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/19577493-09e9-42e3-a2a5-fbdf0c57cb96_isrForCrosstalkSources_855808_49.stderr
parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: <AppFuture at 0x154463b58590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 117 has memoization hash fccce59df34e8bb0eb62f7d8263445f9
parsl.dataflow.memoization INFO: Task 117 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118
parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/9625ce93-8488-4d44-9237-fe1a1c915874_isrForCrosstalkSources_855716_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/9625ce93-8488-4d44-9237-fe1a1c915874_isrForCrosstalkSources_855716_40.stderr
parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: <AppFuture at 0x154463b563d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 081eda5320d7a4d75f7987854b7e852c
parsl.dataflow.memoization INFO: Task 118 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119
parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/661895a0-2546-492c-b5a1-2e6e9e86916e_isrForCrosstalkSources_855810_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/661895a0-2546-492c-b5a1-2e6e9e86916e_isrForCrosstalkSources_855810_21.stderr
parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: <AppFuture at 0x154463b36590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 438c6b9374586a6b6f53de83d0eb7655
parsl.dataflow.memoization INFO: Task 119 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120
parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/13183ba2-a4c2-4303-ab87-ad089230dad4_isrForCrosstalkSources_855808_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/13183ba2-a4c2-4303-ab87-ad089230dad4_isrForCrosstalkSources_855808_25.stderr
parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: <AppFuture at 0x154463b59f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 4e743309905e25f4c32c626081de6580
parsl.dataflow.memoization INFO: Task 120 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121
parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/58068611-5dd0-4205-9817-cd18c9828ab4_isrForCrosstalkSources_855717_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/58068611-5dd0-4205-9817-cd18c9828ab4_isrForCrosstalkSources_855717_50.stderr
parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: <AppFuture at 0x1544637726d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 430c91b206f5fc3f5cb4b501b2c43099
parsl.dataflow.memoization INFO: Task 121 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122
parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/96729785-2aa3-4b32-9c4e-7ac111d64147_isrForCrosstalkSources_855716_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/96729785-2aa3-4b32-9c4e-7ac111d64147_isrForCrosstalkSources_855716_51.stderr
parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: <AppFuture at 0x154463772dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 433c6a032f33f12a31b1d22424e22982
parsl.dataflow.memoization INFO: Task 122 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123
parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/16652e05-e1fb-42c1-928b-8d01e04dfcf7_isrForCrosstalkSources_855717_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/16652e05-e1fb-42c1-928b-8d01e04dfcf7_isrForCrosstalkSources_855717_38.stderr
parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: <AppFuture at 0x154463771890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 123 has memoization hash b5048795fd4fbbec8f4bfe80360b5d18
parsl.dataflow.memoization INFO: Task 123 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124
parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/b4a414a0-7d01-42a0-b74b-06f7d45105de_isrForCrosstalkSources_855718_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/b4a414a0-7d01-42a0-b74b-06f7d45105de_isrForCrosstalkSources_855718_4.stderr
parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: <AppFuture at 0x15446e16cc50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 93ef0cfed59e09af0b06cf019796182a
parsl.dataflow.memoization INFO: Task 124 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125
parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/e3acbd45-b9e9-46d4-937a-45115e8f3439_isrForCrosstalkSources_855809_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/e3acbd45-b9e9-46d4-937a-45115e8f3439_isrForCrosstalkSources_855809_54.stderr
parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: <AppFuture at 0x15446e16c910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 125 has memoization hash cb0a0c84107f30d175320c6987e8e980
parsl.dataflow.memoization INFO: Task 125 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126
parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/fad1697e-7e1b-4c67-ae63-70eabb4d5f9a_isrForCrosstalkSources_855808_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/fad1697e-7e1b-4c67-ae63-70eabb4d5f9a_isrForCrosstalkSources_855808_35.stderr
parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: <AppFuture at 0x154463b36410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 126 has memoization hash f819d96da9143bb8b078c55c0e8eadaf
parsl.dataflow.memoization INFO: Task 126 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127
parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/6c156c9d-50fd-438c-9c65-bb9d4783169d_isrForCrosstalkSources_855717_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/6c156c9d-50fd-438c-9c65-bb9d4783169d_isrForCrosstalkSources_855717_55.stderr
parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: <AppFuture at 0x15446e16fa90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 5e813724ee19cfbda7a6df3f08f91493
parsl.dataflow.memoization INFO: Task 127 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128
parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/02039795-e35e-49d3-b6cc-110edb65347e_isrForCrosstalkSources_855718_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/02039795-e35e-49d3-b6cc-110edb65347e_isrForCrosstalkSources_855718_40.stderr
parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: <AppFuture at 0x1544639f46d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 128 has memoization hash c0aeb4d2c2faa877a9e9f9c8c46b4eb6
parsl.dataflow.memoization INFO: Task 128 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129
parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/7491b2a2-3853-490f-a983-3f040fff7a85_isrForCrosstalkSources_855809_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/7491b2a2-3853-490f-a983-3f040fff7a85_isrForCrosstalkSources_855809_50.stderr
parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: <AppFuture at 0x1544639f5c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 2ed64ed7b1715311c05571f19e93d0cb
parsl.dataflow.memoization INFO: Task 129 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130
parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/0c5c9f28-21b0-4d99-806c-8db222147e56_isrForCrosstalkSources_855810_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/0c5c9f28-21b0-4d99-806c-8db222147e56_isrForCrosstalkSources_855810_51.stderr
parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: <AppFuture at 0x1544639f6190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 130 has memoization hash d50136b6064257b03b39afa0147dc24b
parsl.dataflow.memoization INFO: Task 130 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131
parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/fe8eeecf-4769-477b-942a-6b4e2c14af0f_isrForCrosstalkSources_855718_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/fe8eeecf-4769-477b-942a-6b4e2c14af0f_isrForCrosstalkSources_855718_14.stderr
parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: <AppFuture at 0x1544639f6d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 131 has memoization hash da57276457eb570560c6ab872af27dec
parsl.dataflow.memoization INFO: Task 131 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132
parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/4cbe993e-140f-4030-9ebb-aa7d5e4b2994_isrForCrosstalkSources_855810_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/4cbe993e-140f-4030-9ebb-aa7d5e4b2994_isrForCrosstalkSources_855810_27.stderr
parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: <AppFuture at 0x1544639f5d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 5aec14b6fa2c2c58f7493145582fa284
parsl.dataflow.memoization INFO: Task 132 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133
parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/92aaaa83-5a7f-4997-a00a-7f67bb0c5e4f_isrForCrosstalkSources_855810_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/92aaaa83-5a7f-4997-a00a-7f67bb0c5e4f_isrForCrosstalkSources_855810_49.stderr
parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: <AppFuture at 0x15446371f850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 133 has memoization hash a9c8f525f3b8285ea60d896312ce4f9c
parsl.dataflow.memoization INFO: Task 133 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134
parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/7936f9e0-a682-4c0e-8e3b-7d430745429e_isrForCrosstalkSources_855808_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/7936f9e0-a682-4c0e-8e3b-7d430745429e_isrForCrosstalkSources_855808_17.stderr
parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: <AppFuture at 0x15446371e7d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 76b290dd50b4b0667293990c95421543
parsl.dataflow.memoization INFO: Task 134 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135
parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/ef62a15e-e513-4077-88ed-a04b7a347f3d_isrForCrosstalkSources_855810_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/ef62a15e-e513-4077-88ed-a04b7a347f3d_isrForCrosstalkSources_855810_25.stderr
parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: <AppFuture at 0x15446371dbd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 135 has memoization hash dbb0ecccab5e21d272a7aef0dae1e8ca
parsl.dataflow.memoization INFO: Task 135 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136
parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/70381a29-fdf4-417f-9c55-06f0cb1aef2a_isrForCrosstalkSources_855717_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/70381a29-fdf4-417f-9c55-06f0cb1aef2a_isrForCrosstalkSources_855717_3.stderr
parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: <AppFuture at 0x15446371c0d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 136 has memoization hash f69cde095f1a847735e538ed8fc3302f
parsl.dataflow.memoization INFO: Task 136 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137
parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/54a36252-084f-441f-8bc7-0ebb718f7a35_isrForCrosstalkSources_855718_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/54a36252-084f-441f-8bc7-0ebb718f7a35_isrForCrosstalkSources_855718_47.stderr
parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: <AppFuture at 0x15446389dc90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 91b7b0aea764e8ac0ff4c7840bab59b1
parsl.dataflow.memoization INFO: Task 137 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138
parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/87587180-3ff6-447c-8e6c-432a34d74741_isrForCrosstalkSources_855808_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/87587180-3ff6-447c-8e6c-432a34d74741_isrForCrosstalkSources_855808_19.stderr
parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: <AppFuture at 0x15446389dd90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 138 has memoization hash b0869e8fa47f97149e53fabc370427ed
parsl.dataflow.memoization INFO: Task 138 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139
parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/6fd02e9c-e9f0-4f0f-aaf3-229f004e7ab1_isrForCrosstalkSources_855718_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/6fd02e9c-e9f0-4f0f-aaf3-229f004e7ab1_isrForCrosstalkSources_855718_16.stderr
parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: <AppFuture at 0x15446389ea50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 233bf38ba5b6cfe2ae397db18b6aeeff
parsl.dataflow.memoization INFO: Task 139 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140
parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/07b61f58-1270-43fd-8f77-c87ad54beb5e_isrForCrosstalkSources_855808_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/07b61f58-1270-43fd-8f77-c87ad54beb5e_isrForCrosstalkSources_855808_20.stderr
parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: <AppFuture at 0x154463a1d950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 425184cc0449a6e8cf1e22d49b165f3d
parsl.dataflow.memoization INFO: Task 140 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141
parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/a8499962-f6a5-40c2-a1ae-36336f8ee50e_isrForCrosstalkSources_855809_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/a8499962-f6a5-40c2-a1ae-36336f8ee50e_isrForCrosstalkSources_855809_44.stderr
parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: <AppFuture at 0x1544637764d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 141 has memoization hash d2cc75d0c2517f1551125258a30bf47b
parsl.dataflow.memoization INFO: Task 141 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142
parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/0b291b5c-9f18-4bf8-a2b1-122cbcfd9b8a_isrForCrosstalkSources_855808_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/0b291b5c-9f18-4bf8-a2b1-122cbcfd9b8a_isrForCrosstalkSources_855808_22.stderr
parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: <AppFuture at 0x1544637763d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 9d6aabb509d9742e27b02f1ef4b7f9d6
parsl.dataflow.memoization INFO: Task 142 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143
parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/5e69a2d3-0f93-4f86-a3f5-b2b62b6cb70a_isrForCrosstalkSources_855716_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/5e69a2d3-0f93-4f86-a3f5-b2b62b6cb70a_isrForCrosstalkSources_855716_2.stderr
parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: <AppFuture at 0x1544637741d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 88de254df956f0f5213d266c4e92ad23
parsl.dataflow.memoization INFO: Task 143 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144
parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/f17410d8-1f77-44ab-8591-4d48725d35cb_isrForCrosstalkSources_855808_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/f17410d8-1f77-44ab-8591-4d48725d35cb_isrForCrosstalkSources_855808_62.stderr
parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: <AppFuture at 0x154463776a50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 15899c6a90bed419dfb2a8aab2524e82
parsl.dataflow.memoization INFO: Task 144 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145
parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/df2105a6-d915-46d4-a931-606ad282fc8a_isrForCrosstalkSources_855718_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/df2105a6-d915-46d4-a931-606ad282fc8a_isrForCrosstalkSources_855718_55.stderr
parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: <AppFuture at 0x154463774a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 63accfd11cb6a2e12435ca27c6b08a93
parsl.dataflow.memoization INFO: Task 145 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146
parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/7babea6b-be5c-4194-a09a-d13c725006a6_isrForCrosstalkSources_855808_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/7babea6b-be5c-4194-a09a-d13c725006a6_isrForCrosstalkSources_855808_59.stderr
parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: <AppFuture at 0x1544637856d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 146 has memoization hash fdc9e8384c86cb180536c51e9428d5e6
parsl.dataflow.memoization INFO: Task 146 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147
parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/e60af1bb-60fb-43b3-82f0-019093f39d95_isrForCrosstalkSources_855716_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/e60af1bb-60fb-43b3-82f0-019093f39d95_isrForCrosstalkSources_855716_49.stderr
parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: <AppFuture at 0x154463784f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 147 has memoization hash e3bb14ecde40c9594f7feddf65f092a5
parsl.dataflow.memoization INFO: Task 147 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148
parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/c1ad64db-fd5f-4067-82d7-0585c65ee013_isrForCrosstalkSources_855716_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/c1ad64db-fd5f-4067-82d7-0585c65ee013_isrForCrosstalkSources_855716_39.stderr
parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: <AppFuture at 0x154463785b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 148 has memoization hash a5c8a315c14e8d319dfe24a6fdd285b7
parsl.dataflow.memoization INFO: Task 148 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149
parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/e519861e-e03b-48d2-8acc-28cb6813a4b5_isrForCrosstalkSources_855809_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/e519861e-e03b-48d2-8acc-28cb6813a4b5_isrForCrosstalkSources_855809_43.stderr
parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: <AppFuture at 0x1544637879d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 149 has memoization hash 54782de9b718424e07cc4b0703116af8
parsl.dataflow.memoization INFO: Task 149 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150
parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/38d7fc6e-4ece-4ad8-b1d3-880e22d84cb4_isrForCrosstalkSources_855718_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/38d7fc6e-4ece-4ad8-b1d3-880e22d84cb4_isrForCrosstalkSources_855718_57.stderr
parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: <AppFuture at 0x154463784050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 150 has memoization hash a763c561be28146cebbbc43ddac4efcb
parsl.dataflow.memoization INFO: Task 150 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151
parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/069099a5-4bd1-41e7-a34e-41d99959dc4e_isrForCrosstalkSources_855718_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/069099a5-4bd1-41e7-a34e-41d99959dc4e_isrForCrosstalkSources_855718_52.stderr
parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: <AppFuture at 0x154462205310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 151 has memoization hash f7158b2aa2cac36836948cd56cf3e0db
parsl.dataflow.memoization INFO: Task 151 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152
parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/5ed13e69-7823-499c-a3d6-8e7149317692_isrForCrosstalkSources_855717_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/5ed13e69-7823-499c-a3d6-8e7149317692_isrForCrosstalkSources_855717_36.stderr
parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: <AppFuture at 0x154462205050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 152 has memoization hash a137ae9b85a5d27a1aab9607409a2ba2
parsl.dataflow.memoization INFO: Task 152 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153
parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/af6f50f6-e4b6-4715-b815-59a436bf3ab6_isrForCrosstalkSources_855809_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/af6f50f6-e4b6-4715-b815-59a436bf3ab6_isrForCrosstalkSources_855809_2.stderr
parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: <AppFuture at 0x1544622063d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 8ea55f62b82e6676da7399ab9ed4108b
parsl.dataflow.memoization INFO: Task 153 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154
parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/0522d5f0-65ea-42c6-90a4-3d05abd67d59_isrForCrosstalkSources_855716_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/0522d5f0-65ea-42c6-90a4-3d05abd67d59_isrForCrosstalkSources_855716_62.stderr
parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: <AppFuture at 0x154462206090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 154 has memoization hash d7b4582e072880ad9960dc48c0026b93
parsl.dataflow.memoization INFO: Task 154 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155
parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/d7418246-972e-482e-9f80-5a5cf7c1f9ee_isrForCrosstalkSources_855718_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/d7418246-972e-482e-9f80-5a5cf7c1f9ee_isrForCrosstalkSources_855718_53.stderr
parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: <AppFuture at 0x1544622070d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 362eebf85d03198f08d6df846900d656
parsl.dataflow.memoization INFO: Task 155 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156
parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/003d6ff9-a934-4726-8b9a-57d08841626a_isrForCrosstalkSources_855718_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/003d6ff9-a934-4726-8b9a-57d08841626a_isrForCrosstalkSources_855718_7.stderr
parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: <AppFuture at 0x1544622079d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 9d7ddc1051e3099040f485fec307304e
parsl.dataflow.memoization INFO: Task 156 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157
parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/e1b63be9-a798-41d0-8eab-b88f3f5ed4ee_isrForCrosstalkSources_855809_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/e1b63be9-a798-41d0-8eab-b88f3f5ed4ee_isrForCrosstalkSources_855809_31.stderr
parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: <AppFuture at 0x154462207fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 157 has memoization hash d7f6f3b6b092653a69434ff0bfff5dd0
parsl.dataflow.memoization INFO: Task 157 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158
parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/60593ba5-9a00-44d3-8e8c-5d0c4ad8dcfd_isrForCrosstalkSources_855810_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/60593ba5-9a00-44d3-8e8c-5d0c4ad8dcfd_isrForCrosstalkSources_855810_2.stderr
parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: <AppFuture at 0x154463a60350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 521c7e11d4ba57ad61b71faae2d41f51
parsl.dataflow.memoization INFO: Task 158 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159
parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/5f3d79e2-4cea-419a-9217-e7406d8a5ec7_isrForCrosstalkSources_855718_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/5f3d79e2-4cea-419a-9217-e7406d8a5ec7_isrForCrosstalkSources_855718_9.stderr
parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: <AppFuture at 0x154462210cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 98e6281abd7345ff40c1007e49654620
parsl.dataflow.memoization INFO: Task 159 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160
parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/e648df51-ea28-4656-9556-2035d40380c8_isrForCrosstalkSources_855718_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/e648df51-ea28-4656-9556-2035d40380c8_isrForCrosstalkSources_855718_10.stderr
parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: <AppFuture at 0x154462211990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 160 has memoization hash f4ae18ef2e540fb8875e68c10e130116
parsl.dataflow.memoization INFO: Task 160 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161
parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/361350d8-ee74-4901-8707-e04257e00b9f_isrForCrosstalkSources_855717_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/361350d8-ee74-4901-8707-e04257e00b9f_isrForCrosstalkSources_855717_39.stderr
parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: <AppFuture at 0x1544622122d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 161 has memoization hash cef28d19823f7bce6d5477cd0d03fc5f
parsl.dataflow.memoization INFO: Task 161 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544637e0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162
parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/59a28fba-e9c7-4873-a397-145f46f3c80c_isrForCrosstalkSources_855716_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/59a28fba-e9c7-4873-a397-145f46f3c80c_isrForCrosstalkSources_855716_37.stderr
parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: <AppFuture at 0x154462212b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 2e11e46fb9469c3c2019ed67263f204c
parsl.dataflow.memoization INFO: Task 162 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163
parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/48569b30-1dc4-4a76-9b28-9d64457f17fc_isrForCrosstalkSources_855716_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/48569b30-1dc4-4a76-9b28-9d64457f17fc_isrForCrosstalkSources_855716_27.stderr
parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: <AppFuture at 0x1544622121d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 163 has memoization hash a06a6a364a46330e4ddf15d3a69c7c46
parsl.dataflow.memoization INFO: Task 163 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164
parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/122350dc-a15f-4b57-9da0-70f97ff95102_isrForCrosstalkSources_855716_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/122350dc-a15f-4b57-9da0-70f97ff95102_isrForCrosstalkSources_855716_44.stderr
parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: <AppFuture at 0x154462213f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 164 has memoization hash ea584ee5c48fc119ccbc9f27d5b75dd1
parsl.dataflow.memoization INFO: Task 164 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165
parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/e7f20a5c-b75f-4512-8b8c-4008ecccdaf7_isrForCrosstalkSources_855810_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/e7f20a5c-b75f-4512-8b8c-4008ecccdaf7_isrForCrosstalkSources_855810_57.stderr
parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: <AppFuture at 0x154462213bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 17a510e84afd1b9e7113bc84a4605ba0
parsl.dataflow.memoization INFO: Task 165 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166
parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/5d81e525-ad85-42c7-9608-4f46af9a757c_isrForCrosstalkSources_855810_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/5d81e525-ad85-42c7-9608-4f46af9a757c_isrForCrosstalkSources_855810_18.stderr
parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: <AppFuture at 0x154462204e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 166 has memoization hash f4adfc1013f0f12252939d348b70c2f8
parsl.dataflow.memoization INFO: Task 166 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167
parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/350e2e1e-bf22-44e7-a3e7-6011af902c86_isrForCrosstalkSources_855809_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/350e2e1e-bf22-44e7-a3e7-6011af902c86_isrForCrosstalkSources_855809_18.stderr
parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: <AppFuture at 0x154462225d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 316e89bfc1be86236fc59e3339f1b558
parsl.dataflow.memoization INFO: Task 167 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168
parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/7fb19a06-7f24-4afb-9be0-ba015f845bd7_isrForCrosstalkSources_855810_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/7fb19a06-7f24-4afb-9be0-ba015f845bd7_isrForCrosstalkSources_855810_1.stderr
parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: <AppFuture at 0x154462225790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 921b30cac6b55b5b1f1874835c2ce8e1
parsl.dataflow.memoization INFO: Task 168 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169
parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/799d0c18-366e-4a95-a26d-3a9bc49aaf10_isrForCrosstalkSources_855809_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/799d0c18-366e-4a95-a26d-3a9bc49aaf10_isrForCrosstalkSources_855809_17.stderr
parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: <AppFuture at 0x1544622265d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 240fe90bc0be4c87f1a786356f243d8b
parsl.dataflow.memoization INFO: Task 169 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170
parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/3e6ea057-1857-4ad0-9b66-5d391c48954c_isrForCrosstalkSources_855717_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/3e6ea057-1857-4ad0-9b66-5d391c48954c_isrForCrosstalkSources_855717_51.stderr
parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: <AppFuture at 0x154462225e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 170 has memoization hash cf2dee9a2d2688af0400eb4022a1eab8
parsl.dataflow.memoization INFO: Task 170 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171
parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/ce5fdca5-5516-4c06-9cde-a4f94a16a37a_isrForCrosstalkSources_855808_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/ce5fdca5-5516-4c06-9cde-a4f94a16a37a_isrForCrosstalkSources_855808_52.stderr
parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: <AppFuture at 0x154462227790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 171 has memoization hash ec6b1aeccaa90c57f2088ed665ec2530
parsl.dataflow.memoization INFO: Task 171 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172
parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/99b10e52-d06b-44a2-b4b6-1157486b9ea7_isrForCrosstalkSources_855716_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/99b10e52-d06b-44a2-b4b6-1157486b9ea7_isrForCrosstalkSources_855716_34.stderr
parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: <AppFuture at 0x154462227f10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 4f534cd45095148e4781b62cae858e76
parsl.dataflow.memoization INFO: Task 172 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173
parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/abb76991-ffc2-46e5-a630-8c913de49c8d_isrForCrosstalkSources_855809_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/abb76991-ffc2-46e5-a630-8c913de49c8d_isrForCrosstalkSources_855809_7.stderr
parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: <AppFuture at 0x15446222c6d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 10e018a18c26db9b0ba4a408c452ffdc
parsl.dataflow.memoization INFO: Task 173 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174
parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/78be7c6b-f527-4f2c-9b5f-4feba01f7a66_isrForCrosstalkSources_855810_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/78be7c6b-f527-4f2c-9b5f-4feba01f7a66_isrForCrosstalkSources_855810_12.stderr
parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: <AppFuture at 0x15446222c890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 78814ef32036b48919551dd40bcac938
parsl.dataflow.memoization INFO: Task 174 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175
parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/0646698b-954e-49f0-9fda-8962d99cec65_isrForCrosstalkSources_855808_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/0646698b-954e-49f0-9fda-8962d99cec65_isrForCrosstalkSources_855808_36.stderr
parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: <AppFuture at 0x15446222d690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 981fc4919ee5b26dce83655b02b6a366
parsl.dataflow.memoization INFO: Task 175 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176
parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/6080d48e-23e8-4405-964c-3507e295bba9_isrForCrosstalkSources_855809_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/6080d48e-23e8-4405-964c-3507e295bba9_isrForCrosstalkSources_855809_32.stderr
parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: <AppFuture at 0x15446222e110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 5bec02355d4f0f0d21fd4fe5fa3e9ac8
parsl.dataflow.memoization INFO: Task 176 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177
parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/46fb84b8-b959-4285-a793-a85298667fb2_isrForCrosstalkSources_855808_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/46fb84b8-b959-4285-a793-a85298667fb2_isrForCrosstalkSources_855808_42.stderr
parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: <AppFuture at 0x15446222dfd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 177 has memoization hash cdce0c8b4fbce4763f0b159423f7dd50
parsl.dataflow.memoization INFO: Task 177 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178
parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/6b683bc7-2c53-42b8-bb8d-3d428e957617_isrForCrosstalkSources_855716_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/6b683bc7-2c53-42b8-bb8d-3d428e957617_isrForCrosstalkSources_855716_42.stderr
parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: <AppFuture at 0x15446222e990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 178 has memoization hash e4bdfcaa7dbb3dc36e11fde9f5787156
parsl.dataflow.memoization INFO: Task 178 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179
parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/a90e06cb-c3ae-4fdb-ae54-fac83e3d9cf0_isrForCrosstalkSources_855809_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/a90e06cb-c3ae-4fdb-ae54-fac83e3d9cf0_isrForCrosstalkSources_855809_60.stderr
parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: <AppFuture at 0x15446222fa10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 67469de6bf6bb38788bc815630d921d1
parsl.dataflow.memoization INFO: Task 179 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180
parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/c504d0bf-5165-46ba-aff2-1d0919ad5df4_isrForCrosstalkSources_855808_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/c504d0bf-5165-46ba-aff2-1d0919ad5df4_isrForCrosstalkSources_855808_23.stderr
parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: <AppFuture at 0x15446222ee50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 7785eccff112daf35119655d052abb2c
parsl.dataflow.memoization INFO: Task 180 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181
parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/178a81dd-c38f-4e6b-b251-c4090a89d09d_isrForCrosstalkSources_855718_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/178a81dd-c38f-4e6b-b251-c4090a89d09d_isrForCrosstalkSources_855718_48.stderr
parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: <AppFuture at 0x154462211250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 0da8c4fd2d8d0c1b54668f6de5948f85
parsl.dataflow.memoization INFO: Task 181 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182
parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/fa4fb907-6ce3-4456-932c-c6deaab9190e_isrForCrosstalkSources_855808_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/fa4fb907-6ce3-4456-932c-c6deaab9190e_isrForCrosstalkSources_855808_5.stderr
parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: <AppFuture at 0x1544622394d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 182 has memoization hash ef15d30c222b04dd773d728538be9acc
parsl.dataflow.memoization INFO: Task 182 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183
parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/7ed4f035-ea91-4811-bc9b-12d39caefb58_isrForCrosstalkSources_855810_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/7ed4f035-ea91-4811-bc9b-12d39caefb58_isrForCrosstalkSources_855810_35.stderr
parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: <AppFuture at 0x154462239350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 4c22e600c8312961e2c5ae0f9dda33c5
parsl.dataflow.memoization INFO: Task 183 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184
parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/eccd42c5-acaa-47f7-84f9-c73289cdcc16_isrForCrosstalkSources_855716_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/eccd42c5-acaa-47f7-84f9-c73289cdcc16_isrForCrosstalkSources_855716_22.stderr
parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: <AppFuture at 0x15446223a550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 8c78d1333752080a6f0e8fd3f7d5197d
parsl.dataflow.memoization INFO: Task 184 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185
parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/5e1baa19-8b47-4d03-8a4a-8003677ca3da_isrForCrosstalkSources_855810_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/5e1baa19-8b47-4d03-8a4a-8003677ca3da_isrForCrosstalkSources_855810_59.stderr
parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: <AppFuture at 0x15446223a8d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 185 has memoization hash bebfbf96db2bc3a10dc3f13ff069a4b4
parsl.dataflow.memoization INFO: Task 185 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186
parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/65ff7485-d7c9-414e-a4fb-44696a46c982_isrForCrosstalkSources_855809_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/65ff7485-d7c9-414e-a4fb-44696a46c982_isrForCrosstalkSources_855809_11.stderr
parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 186 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: <AppFuture at 0x15446223b690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 8fc70fcac76d009ed1834d68aecfd750
parsl.dataflow.memoization INFO: Task 186 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 187
parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/135bda14-ad3e-4f00-8f04-267c02ae2c0e_isrForCrosstalkSources_855716_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/135bda14-ad3e-4f00-8f04-267c02ae2c0e_isrForCrosstalkSources_855716_59.stderr
parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 187 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: <AppFuture at 0x15446223bd10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 187 has memoization hash 0e65dc8c3ebcfe74977bbb7ab3be3ca8
parsl.dataflow.memoization INFO: Task 187 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 188
parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/7d03ab40-ed2d-4054-9de4-09670b728956_isrForCrosstalkSources_855808_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/7d03ab40-ed2d-4054-9de4-09670b728956_isrForCrosstalkSources_855808_21.stderr
parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 188 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: <AppFuture at 0x154463bf1510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 188 has memoization hash b431d8f7878b4ec3f9defa295e18db13
parsl.dataflow.memoization INFO: Task 188 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 189
parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/dbb3e4c7-bae6-4736-930b-b80a686c9a35_isrForCrosstalkSources_855808_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/dbb3e4c7-bae6-4736-930b-b80a686c9a35_isrForCrosstalkSources_855808_57.stderr
parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 189 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: <AppFuture at 0x154462240a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 189 has memoization hash e0812e908b155464d95b32459014f768
parsl.dataflow.memoization INFO: Task 189 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 190
parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/2080af5f-882e-4a6d-890e-2856bee360b0_isrForCrosstalkSources_855716_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/2080af5f-882e-4a6d-890e-2856bee360b0_isrForCrosstalkSources_855716_23.stderr
parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 190 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: <AppFuture at 0x154462241710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 9bede29c71f2b5865a899ce6b7ed7e72
parsl.dataflow.memoization INFO: Task 190 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 191
parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/b48aaa06-fcd5-41b5-a2e6-5aaecda8de4f_isrForCrosstalkSources_855718_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/b48aaa06-fcd5-41b5-a2e6-5aaecda8de4f_isrForCrosstalkSources_855718_62.stderr
parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 191 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: <AppFuture at 0x154462242ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 80511578bcfe1d9828cc22aac93b18f1
parsl.dataflow.memoization INFO: Task 191 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 192
parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/a026161d-2297-48e6-81f9-f39b24cce4cf_isrForCrosstalkSources_855810_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/a026161d-2297-48e6-81f9-f39b24cce4cf_isrForCrosstalkSources_855810_15.stderr
parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 192 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: <AppFuture at 0x154463b4aa10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 192 has memoization hash f34dc1199e3b0b52bcbcc712a7c30a10
parsl.dataflow.memoization INFO: Task 192 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 193
parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/5b29b87d-d67b-4f83-8cc9-f8192a0c37fe_isrForCrosstalkSources_855718_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/5b29b87d-d67b-4f83-8cc9-f8192a0c37fe_isrForCrosstalkSources_855718_3.stderr
parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 193 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: <AppFuture at 0x1544622436d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 193 has memoization hash f62f2bb5d4f536655bd45bf2a1f43e3f
parsl.dataflow.memoization INFO: Task 193 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 194
parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/e31dffa0-a6bc-41c9-a4c0-d63d52d1c5ac_isrForCrosstalkSources_855810_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/e31dffa0-a6bc-41c9-a4c0-d63d52d1c5ac_isrForCrosstalkSources_855810_28.stderr
parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 194 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: <AppFuture at 0x1544622430d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 194 has memoization hash c334031620255acd9eb74e1b283a8d24
parsl.dataflow.memoization INFO: Task 194 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 195
parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/1857a2e6-37c5-45ae-ac61-ac17eff4aebd_isrForCrosstalkSources_855718_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/1857a2e6-37c5-45ae-ac61-ac17eff4aebd_isrForCrosstalkSources_855718_6.stderr
parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 195 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: <AppFuture at 0x154462242710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 12757c7e988448e1912eb07ef1e661d4
parsl.dataflow.memoization INFO: Task 195 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 196
parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/a145f8e5-73ad-4559-9599-725daec5614a_isrForCrosstalkSources_855809_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/a145f8e5-73ad-4559-9599-725daec5614a_isrForCrosstalkSources_855809_35.stderr
parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 196 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: <AppFuture at 0x15446222c810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 196 has memoization hash a89abac441908b032eb8e932c2f433d6
parsl.dataflow.memoization INFO: Task 196 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 197
parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/e0882b93-5cd9-4dfd-ba5f-a4fedcbc2c1c_isrForCrosstalkSources_855718_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/e0882b93-5cd9-4dfd-ba5f-a4fedcbc2c1c_isrForCrosstalkSources_855718_49.stderr
parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 197 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: <AppFuture at 0x15446224d950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 197 has memoization hash f5fa84e01fd499469595d51b20b3bc01
parsl.dataflow.memoization INFO: Task 197 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 198
parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/9e758c56-2eb6-45be-b405-de6c26920628_isrForCrosstalkSources_855810_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/9e758c56-2eb6-45be-b405-de6c26920628_isrForCrosstalkSources_855810_30.stderr
parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 198 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: <AppFuture at 0x15446224c150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 83698a131a77853123eb319e86c37258
parsl.dataflow.memoization INFO: Task 198 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 199
parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/9881421b-55b5-469e-8422-fe0f7ab78f2d_isrForCrosstalkSources_855808_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/9881421b-55b5-469e-8422-fe0f7ab78f2d_isrForCrosstalkSources_855808_37.stderr
parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 199 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: <AppFuture at 0x15446224e550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 199 has memoization hash edb021ce91ccffb510e0c0b4f977aaa3
parsl.dataflow.memoization INFO: Task 199 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 200
parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/16b1ec3d-4433-41e8-b231-86e6b48aed09_isrForCrosstalkSources_855717_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/16b1ec3d-4433-41e8-b231-86e6b48aed09_isrForCrosstalkSources_855717_40.stderr
parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 200 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: <AppFuture at 0x15446224ee10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 200 has memoization hash f9822fffdcebe75ad49a8da6c24e7e4c
parsl.dataflow.memoization INFO: Task 200 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 201
parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/bb75a2d7-b156-4b60-a12d-2f167e50792b_isrForCrosstalkSources_855808_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/bb75a2d7-b156-4b60-a12d-2f167e50792b_isrForCrosstalkSources_855808_34.stderr
parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 201 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: <AppFuture at 0x15446224fd90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 984698830d50fd3fb0b0c1308520115a
parsl.dataflow.memoization INFO: Task 201 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 202
parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/c9e0836b-0b17-4787-84ed-c0e1332b0ff3_isrForCrosstalkSources_855809_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/c9e0836b-0b17-4787-84ed-c0e1332b0ff3_isrForCrosstalkSources_855809_25.stderr
parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 202 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: <AppFuture at 0x15446224f990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 1bd93d8cb9787162de16ffade4616913
parsl.dataflow.memoization INFO: Task 202 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 203
parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/c8799da8-3cd6-4a16-bbb5-6799ff868388_isrForCrosstalkSources_855809_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/c8799da8-3cd6-4a16-bbb5-6799ff868388_isrForCrosstalkSources_855809_46.stderr
parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 203 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: <AppFuture at 0x1544622112d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 6652b7e29e94530345e7db7ce0820a4e
parsl.dataflow.memoization INFO: Task 203 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 204
parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/984f5f95-1514-4f98-bd51-c4d39048f787_isrForCrosstalkSources_855717_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/984f5f95-1514-4f98-bd51-c4d39048f787_isrForCrosstalkSources_855717_17.stderr
parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 204 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: <AppFuture at 0x154462258210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 4cef4df281c047b87cedaf0229d61082
parsl.dataflow.memoization INFO: Task 204 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 205
parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/3369a009-484e-419e-b499-76dbffc4bf77_isrForCrosstalkSources_855717_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/3369a009-484e-419e-b499-76dbffc4bf77_isrForCrosstalkSources_855717_43.stderr
parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 205 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: <AppFuture at 0x1544622595d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 72af6055041e37385ca6a9150e5af222
parsl.dataflow.memoization INFO: Task 205 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 206
parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/52ae9c61-1023-4d17-b911-11ce77ba23b0_isrForCrosstalkSources_855810_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/52ae9c61-1023-4d17-b911-11ce77ba23b0_isrForCrosstalkSources_855810_44.stderr
parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 206 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: <AppFuture at 0x15446225a350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 206 has memoization hash c99f35b7aeb6612b8677bf3de600eee3
parsl.dataflow.memoization INFO: Task 206 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 207
parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/a1247340-9c34-4a2e-bbf3-c06eb68b02e5_isrForCrosstalkSources_855717_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/a1247340-9c34-4a2e-bbf3-c06eb68b02e5_isrForCrosstalkSources_855717_59.stderr
parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 207 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: <AppFuture at 0x15446225a810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 207 has memoization hash eb039b3137cc46c88b56e111a4732ca6
parsl.dataflow.memoization INFO: Task 207 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 208
parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/99fa8782-0c6c-4ac4-9a22-078474c96644_isrForCrosstalkSources_855810_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/99fa8782-0c6c-4ac4-9a22-078474c96644_isrForCrosstalkSources_855810_4.stderr
parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 208 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: <AppFuture at 0x15446225b150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 182e02df5cce2846203c1dcac822bd15
parsl.dataflow.memoization INFO: Task 208 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 209
parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/c969b9c3-6083-486a-ab80-81e8c09bb908_isrForCrosstalkSources_855808_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/c969b9c3-6083-486a-ab80-81e8c09bb908_isrForCrosstalkSources_855808_28.stderr
parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 209 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: <AppFuture at 0x15446225aa50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 209 has memoization hash ca6a6f668281c7147cccf5d29035a657
parsl.dataflow.memoization INFO: Task 209 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 210
parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/9ac6ffd9-99c3-4dc2-a98c-4de28a43e654_isrForCrosstalkSources_855717_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/9ac6ffd9-99c3-4dc2-a98c-4de28a43e654_isrForCrosstalkSources_855717_2.stderr
parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 210 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: <AppFuture at 0x15446225b890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 7f3ac8a44ec72b3696f7e1c52d7ca6b6
parsl.dataflow.memoization INFO: Task 210 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 211
parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/a855c5c2-ec39-4a78-9618-7d610818d0a7_isrForCrosstalkSources_855808_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/a855c5c2-ec39-4a78-9618-7d610818d0a7_isrForCrosstalkSources_855808_51.stderr
parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 211 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: <AppFuture at 0x154462260090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 211 has memoization hash cb92ef0182d9dfc289662eecb1469b20
parsl.dataflow.memoization INFO: Task 211 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 212
parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/941a7d7d-ee05-44d6-aafc-0aa32f40d5c0_isrForCrosstalkSources_855808_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/941a7d7d-ee05-44d6-aafc-0aa32f40d5c0_isrForCrosstalkSources_855808_48.stderr
parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 212 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: <AppFuture at 0x154462262110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 212 has memoization hash faa57f50b2c69ce662d003e0be512b06
parsl.dataflow.memoization INFO: Task 212 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 213
parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/393d6ce4-206e-4fd9-9c03-6495c59efdc2_isrForCrosstalkSources_855718_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/393d6ce4-206e-4fd9-9c03-6495c59efdc2_isrForCrosstalkSources_855718_43.stderr
parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 213 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: <AppFuture at 0x154462260f10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 360b6f9c8872a057148060f77ebcdeaf
parsl.dataflow.memoization INFO: Task 213 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 214
parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/5b33f8a7-a8ce-489b-b51c-0755aed1b7ab_isrForCrosstalkSources_855808_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/5b33f8a7-a8ce-489b-b51c-0755aed1b7ab_isrForCrosstalkSources_855808_45.stderr
parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 214 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: <AppFuture at 0x1544622621d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 214 has memoization hash 78745bb48b76168ba1885f3be067f698
parsl.dataflow.memoization INFO: Task 214 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 215
parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/9ace02a0-85cf-4596-8726-61d2c5cc37c6_isrForCrosstalkSources_855808_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/9ace02a0-85cf-4596-8726-61d2c5cc37c6_isrForCrosstalkSources_855808_16.stderr
parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 215 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: <AppFuture at 0x1544622625d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 215 has memoization hash fd2336b5816e7f56c3261b6c1cea2c20
parsl.dataflow.memoization INFO: Task 215 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 216
parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/edd140f5-cb14-44d1-aad4-6494c6830de3_isrForCrosstalkSources_855809_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/edd140f5-cb14-44d1-aad4-6494c6830de3_isrForCrosstalkSources_855809_21.stderr
parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 216 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: <AppFuture at 0x154462263410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 216 has memoization hash 7ebcf9f5818bbe5733383c0eeda8f477
parsl.dataflow.memoization INFO: Task 216 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 217
parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/1a1f3922-8fa1-45ff-a6b7-ce85175a8cb0_isrForCrosstalkSources_855717_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/1a1f3922-8fa1-45ff-a6b7-ce85175a8cb0_isrForCrosstalkSources_855717_34.stderr
parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 217 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: <AppFuture at 0x154462263e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 2b33e545bb1ba69f9ca7387c4f79d43f
parsl.dataflow.memoization INFO: Task 217 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 218
parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/24daf74d-17fb-426f-b223-4fcc94ace12f_isrForCrosstalkSources_855716_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/24daf74d-17fb-426f-b223-4fcc94ace12f_isrForCrosstalkSources_855716_9.stderr
parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 218 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: <AppFuture at 0x154462241410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 6caefd252b34cb9b4a7f33148d5992c0
parsl.dataflow.memoization INFO: Task 218 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 219
parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/401dfa2d-9d29-4e35-86a6-ca3d24b99b03_isrForCrosstalkSources_855718_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/401dfa2d-9d29-4e35-86a6-ca3d24b99b03_isrForCrosstalkSources_855718_33.stderr
parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 219 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: <AppFuture at 0x154462268fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 4cfc427cc7d8426caf7bad7c0ea13362
parsl.dataflow.memoization INFO: Task 219 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 220
parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/3d4c6612-d6b1-45bf-af3d-139be4f82eb4_isrForCrosstalkSources_855808_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/3d4c6612-d6b1-45bf-af3d-139be4f82eb4_isrForCrosstalkSources_855808_50.stderr
parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 220 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: <AppFuture at 0x1544622697d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 02b8749ec7d60afb561d50ad22c77f6b
parsl.dataflow.memoization INFO: Task 220 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 221
parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/7fbf74d1-8679-4d5d-a5c9-0416c7e689ba_isrForCrosstalkSources_855809_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/7fbf74d1-8679-4d5d-a5c9-0416c7e689ba_isrForCrosstalkSources_855809_62.stderr
parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 221 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: <AppFuture at 0x154462269d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 221 has memoization hash f7822698e80c0a0fe1ea928233253acb
parsl.dataflow.memoization INFO: Task 221 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 222
parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/888f8c68-5adf-471b-b46a-aaf791f8e3c9_isrForCrosstalkSources_855717_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/888f8c68-5adf-471b-b46a-aaf791f8e3c9_isrForCrosstalkSources_855717_60.stderr
parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 222 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: <AppFuture at 0x15446226add0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 222 has memoization hash 4ce39d92b292ec4b48357435104bef8b
parsl.dataflow.memoization INFO: Task 222 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 223
parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/ac49009b-fd4b-4826-b880-6d3d7c263368_isrForCrosstalkSources_855717_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/ac49009b-fd4b-4826-b880-6d3d7c263368_isrForCrosstalkSources_855717_41.stderr
parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 223 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: <AppFuture at 0x15446226a850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 8d64b18f185b6e88a55f6caaa6960855
parsl.dataflow.memoization INFO: Task 223 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 224
parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/efa72da9-6cd2-4959-863c-5f4da95e39ca_isrForCrosstalkSources_855810_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/efa72da9-6cd2-4959-863c-5f4da95e39ca_isrForCrosstalkSources_855810_9.stderr
parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 224 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: <AppFuture at 0x15446226bdd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 224 has memoization hash b835d2b75d071fee32cf933d9ca419d0
parsl.dataflow.memoization INFO: Task 224 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 225
parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/102f6582-fd94-4c8c-8709-a958020e786a_isrForCrosstalkSources_855717_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/102f6582-fd94-4c8c-8709-a958020e786a_isrForCrosstalkSources_855717_62.stderr
parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 225 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: <AppFuture at 0x154462224110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 225 has memoization hash d0d8fcbd6358dcd8300530fa4fcace90
parsl.dataflow.memoization INFO: Task 225 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 226
parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/31c9cb45-e8b6-4e36-acd6-cf11b8202314_isrForCrosstalkSources_855718_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/31c9cb45-e8b6-4e36-acd6-cf11b8202314_isrForCrosstalkSources_855718_20.stderr
parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 226 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: <AppFuture at 0x154462275290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 226 has memoization hash 50a0273c82e6c26c0fcff82f50891ae8
parsl.dataflow.memoization INFO: Task 226 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 227
parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/a4a37ddc-61f5-42af-8c98-c99dfaaf6f32_isrForCrosstalkSources_855717_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/a4a37ddc-61f5-42af-8c98-c99dfaaf6f32_isrForCrosstalkSources_855717_47.stderr
parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 227 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: <AppFuture at 0x154462275910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 227 has memoization hash 12328fa5f160127db85e8f191bb5f6e8
parsl.dataflow.memoization INFO: Task 227 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 228
parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/005d275e-e745-4145-957b-91a9cfa63aaf_isrForCrosstalkSources_855717_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/005d275e-e745-4145-957b-91a9cfa63aaf_isrForCrosstalkSources_855717_14.stderr
parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 228 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: <AppFuture at 0x1544622750d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 228 has memoization hash 2b77af1578ad1a650015aa90b34aef6d
parsl.dataflow.memoization INFO: Task 228 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 229
parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/8ba77c77-1557-414d-9cd3-67707fcd7a51_isrForCrosstalkSources_855718_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/8ba77c77-1557-414d-9cd3-67707fcd7a51_isrForCrosstalkSources_855718_23.stderr
parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 229 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: <AppFuture at 0x154462276450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 229 has memoization hash d215b1cb9081473e603cb862f9ae79b7
parsl.dataflow.memoization INFO: Task 229 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 230
parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/9b108f64-3ff0-40e3-a2d7-56a4a54d4d9f_isrForCrosstalkSources_855808_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/9b108f64-3ff0-40e3-a2d7-56a4a54d4d9f_isrForCrosstalkSources_855808_24.stderr
parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 230 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: <AppFuture at 0x154462276c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 230 has memoization hash d7ec21c67296ca49cf9cf5e9929b9581
parsl.dataflow.memoization INFO: Task 230 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 231
parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/a971e37d-e898-438b-832c-b30cb2c10066_isrForCrosstalkSources_855716_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/a971e37d-e898-438b-832c-b30cb2c10066_isrForCrosstalkSources_855716_28.stderr
parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 231 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: <AppFuture at 0x154462277a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 231 has memoization hash 56f2beae56dd12a8cbebe7d60d779aaa
parsl.dataflow.memoization INFO: Task 231 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 232
parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/da3efb51-9f28-4749-88f3-49fcda3e126c_isrForCrosstalkSources_855810_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/da3efb51-9f28-4749-88f3-49fcda3e126c_isrForCrosstalkSources_855810_14.stderr
parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 232 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: <AppFuture at 0x154462277390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 3159b3273378281d5f382e5b7b3d3394
parsl.dataflow.memoization INFO: Task 232 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 233
parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/3688284c-4bec-4b9e-a7af-7bf3a93917f0_isrForCrosstalkSources_855809_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/3688284c-4bec-4b9e-a7af-7bf3a93917f0_isrForCrosstalkSources_855809_30.stderr
parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 233 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: <AppFuture at 0x154462268e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 233 has memoization hash 8b83c68b44a0aaf2bbb92cdceb1e9045
parsl.dataflow.memoization INFO: Task 233 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 234
parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/4389265c-f505-4f6b-bb57-f9a5da7c598a_isrForCrosstalkSources_855809_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/4389265c-f505-4f6b-bb57-f9a5da7c598a_isrForCrosstalkSources_855809_59.stderr
parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 234 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: <AppFuture at 0x154462281410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 2b7ae5ae7adf7d7be0cbe7557533fac3
parsl.dataflow.memoization INFO: Task 234 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 235
parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/2c3f4669-3844-4792-8bc4-f330aaba005b_isrForCrosstalkSources_855808_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/2c3f4669-3844-4792-8bc4-f330aaba005b_isrForCrosstalkSources_855808_2.stderr
parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 235 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: <AppFuture at 0x154462280fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 02ed4025361fa8696705c4e271caf2f0
parsl.dataflow.memoization INFO: Task 235 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 236
parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/e67a975e-d97a-45b6-b542-4b0fc1b24f0d_isrForCrosstalkSources_855716_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/e67a975e-d97a-45b6-b542-4b0fc1b24f0d_isrForCrosstalkSources_855716_18.stderr
parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 236 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: <AppFuture at 0x154462282550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 236 has memoization hash c0e5d7573887460b9a62b332a02cd7fe
parsl.dataflow.memoization INFO: Task 236 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 237
parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/cdde492a-f1be-419c-b348-13e4d7a1bfe9_isrForCrosstalkSources_855810_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/cdde492a-f1be-419c-b348-13e4d7a1bfe9_isrForCrosstalkSources_855810_11.stderr
parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 237 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: <AppFuture at 0x154462283150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 128788b48e78e7f542b70caa6743421f
parsl.dataflow.memoization INFO: Task 237 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 238
parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/9f09e316-f43a-4fff-be18-52b8c2314ce7_isrForCrosstalkSources_855809_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/9f09e316-f43a-4fff-be18-52b8c2314ce7_isrForCrosstalkSources_855809_6.stderr
parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 238 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: <AppFuture at 0x154462283090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 139a0e032e86b6a7b84c6200bfe1412a
parsl.dataflow.memoization INFO: Task 238 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 239
parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/e6990b85-d9cd-450f-8f43-558de96da459_isrForCrosstalkSources_855718_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/e6990b85-d9cd-450f-8f43-558de96da459_isrForCrosstalkSources_855718_60.stderr
parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 239 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: <AppFuture at 0x1544622836d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 239 has memoization hash 0fa7501f495a7f37abf134cd9441c4aa
parsl.dataflow.memoization INFO: Task 239 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 240
parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/d44ac05f-a09b-4da1-a689-51c5a8a41fcd_isrForCrosstalkSources_855809_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/d44ac05f-a09b-4da1-a689-51c5a8a41fcd_isrForCrosstalkSources_855809_13.stderr
parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 240 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: <AppFuture at 0x15446224cad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 719d839d94a2c5eef47a82ebc1e6710a
parsl.dataflow.memoization INFO: Task 240 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 241
parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/95c8204b-15fd-443b-b282-152268cd731b_isrForCrosstalkSources_855718_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/95c8204b-15fd-443b-b282-152268cd731b_isrForCrosstalkSources_855718_22.stderr
parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 241 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: <AppFuture at 0x154462284a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 997ff01a901915ce810dea8ad93798bb
parsl.dataflow.memoization INFO: Task 241 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 242
parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/ffd28581-3f9d-4e6e-a69c-7921dd8a0973_isrForCrosstalkSources_855716_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/ffd28581-3f9d-4e6e-a69c-7921dd8a0973_isrForCrosstalkSources_855716_19.stderr
parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 242 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: <AppFuture at 0x154462285d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 242 has memoization hash e8eeb468565663d7562d0eb7c272b766
parsl.dataflow.memoization INFO: Task 242 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 243
parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/40e8b8cd-3d33-4d0d-b7fc-11a1f59b3cae_isrForCrosstalkSources_855717_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/40e8b8cd-3d33-4d0d-b7fc-11a1f59b3cae_isrForCrosstalkSources_855717_31.stderr
parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 243 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: <AppFuture at 0x154462285c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 243 has memoization hash 9a6866419970beb4ced0c5d70f9f093e
parsl.dataflow.memoization INFO: Task 243 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 244
parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/2a7bc524-c03f-481f-9115-b3aab782bc25_isrForCrosstalkSources_855717_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/2a7bc524-c03f-481f-9115-b3aab782bc25_isrForCrosstalkSources_855717_16.stderr
parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 244 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: <AppFuture at 0x154462285c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 263ccaeaf07b3f70ef48df6a6aab9f1c
parsl.dataflow.memoization INFO: Task 244 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 245
parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/7059036c-46a2-46d7-907c-d71ee72eae55_isrForCrosstalkSources_855718_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/7059036c-46a2-46d7-907c-d71ee72eae55_isrForCrosstalkSources_855718_11.stderr
parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 245 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: <AppFuture at 0x154462286910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 245 has memoization hash db033121155852952332113a98f41491
parsl.dataflow.memoization INFO: Task 245 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 246
parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/168911cb-16fc-43c1-ad8b-2a19170ea71f_isrForCrosstalkSources_855810_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/168911cb-16fc-43c1-ad8b-2a19170ea71f_isrForCrosstalkSources_855810_20.stderr
parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 246 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: <AppFuture at 0x1544622875d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 77b9089161cf2c47d669a4a75bfa91a5
parsl.dataflow.memoization INFO: Task 246 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 247
parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/5534f9e9-13fc-48b7-9280-ebd73fa59f67_isrForCrosstalkSources_855809_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/5534f9e9-13fc-48b7-9280-ebd73fa59f67_isrForCrosstalkSources_855809_29.stderr
parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 247 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: <AppFuture at 0x154462287d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 0d51a2e23b48ce6dee27848e50944b11
parsl.dataflow.memoization INFO: Task 247 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 248
parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/a453a728-fc1e-4391-9a9e-07c287b12884_isrForCrosstalkSources_855809_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/a453a728-fc1e-4391-9a9e-07c287b12884_isrForCrosstalkSources_855809_53.stderr
parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 248 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: <AppFuture at 0x154462280250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 248 has memoization hash 4efd9164115527fca491e26dadf0f542
parsl.dataflow.memoization INFO: Task 248 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 249
parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/c31ea1dc-76e9-454b-92d7-50e9b0a58ab0_isrForCrosstalkSources_855718_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/c31ea1dc-76e9-454b-92d7-50e9b0a58ab0_isrForCrosstalkSources_855718_15.stderr
parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 249 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: <AppFuture at 0x154462298f10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 249 has memoization hash 21ef9ace09101e3492217799792947b8
parsl.dataflow.memoization INFO: Task 249 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 250
parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/7e39db8d-6c21-4648-81cc-9298a6e73ae2_isrForCrosstalkSources_855716_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/7e39db8d-6c21-4648-81cc-9298a6e73ae2_isrForCrosstalkSources_855716_12.stderr
parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 250 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: <AppFuture at 0x1544622994d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 250 has memoization hash 8fc1a699f0d9bf639f6ca6cf7fdab1ad
parsl.dataflow.memoization INFO: Task 250 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 251
parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/9743ab27-5fa3-4424-bf77-89fd4c19cc00_isrForCrosstalkSources_855809_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/9743ab27-5fa3-4424-bf77-89fd4c19cc00_isrForCrosstalkSources_855809_45.stderr
parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 251 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: <AppFuture at 0x15446229a550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 251 has memoization hash 323a4c70bc3e59589cf500fd6ae12322
parsl.dataflow.memoization INFO: Task 251 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 252
parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/2ebeaeb1-7efe-429c-9b5d-80779d35b85d_isrForCrosstalkSources_855716_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/2ebeaeb1-7efe-429c-9b5d-80779d35b85d_isrForCrosstalkSources_855716_26.stderr
parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 252 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: <AppFuture at 0x15446229ab10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 252 has memoization hash d27181e60f36f0d2e838762d3ae515b0
parsl.dataflow.memoization INFO: Task 252 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 253
parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/1d17a912-cbb7-4bd9-b336-ddbb556e1e7d_isrForCrosstalkSources_855809_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/1d17a912-cbb7-4bd9-b336-ddbb556e1e7d_isrForCrosstalkSources_855809_27.stderr
parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 253 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: <AppFuture at 0x15446229b350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 2d6ea3cb8a3f8d1dacce730841ebc208
parsl.dataflow.memoization INFO: Task 253 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 254
parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/97585b2f-c575-4912-b0e0-adf3ce82f49f_isrForCrosstalkSources_855718_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/97585b2f-c575-4912-b0e0-adf3ce82f49f_isrForCrosstalkSources_855718_25.stderr
parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 254 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: <AppFuture at 0x15446229b7d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 254 has memoization hash ecd3d9ffe4a0ef41ee3d1b7786500ab8
parsl.dataflow.memoization INFO: Task 254 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 255
parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/bb478b12-7bb5-4b0b-8463-8761244c2cea_isrForCrosstalkSources_855716_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/bb478b12-7bb5-4b0b-8463-8761244c2cea_isrForCrosstalkSources_855716_10.stderr
parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 255 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: <AppFuture at 0x154462298d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 255 has memoization hash d4508bb2bf580a702aaa5221740d1b4e
parsl.dataflow.memoization INFO: Task 255 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 256
parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/60321cae-52e8-405b-8f2e-0442f93caa80_isrForCrosstalkSources_855809_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/60321cae-52e8-405b-8f2e-0442f93caa80_isrForCrosstalkSources_855809_36.stderr
parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 256 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: <AppFuture at 0x1544622a0710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 256 has memoization hash 845de65f493c8d3fda3adc9e97a4c798
parsl.dataflow.memoization INFO: Task 256 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 257
parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/a4f86ebe-87c4-4b2e-9aca-f5ff146cbbee_isrForCrosstalkSources_855716_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/a4f86ebe-87c4-4b2e-9aca-f5ff146cbbee_isrForCrosstalkSources_855716_61.stderr
parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 257 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: <AppFuture at 0x1544622a2110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 257 has memoization hash 91568c03361363d852dfd50672382936
parsl.dataflow.memoization INFO: Task 257 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 258
parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/2e6195ed-c15a-4ee2-861a-d26e040fee73_isrForCrosstalkSources_855808_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/2e6195ed-c15a-4ee2-861a-d26e040fee73_isrForCrosstalkSources_855808_55.stderr
parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 258 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: <AppFuture at 0x1544622a1c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 258 has memoization hash 34f2ac42e99e0a854c4d45a9a248fd5b
parsl.dataflow.memoization INFO: Task 258 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 259
parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/fc16ea36-d4fe-4ad3-badf-06bbb8ba1eba_isrForCrosstalkSources_855809_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/fc16ea36-d4fe-4ad3-badf-06bbb8ba1eba_isrForCrosstalkSources_855809_8.stderr
parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 259 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: <AppFuture at 0x1544622a24d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 59619b3a5044affe6c7ffa32948c2e55
parsl.dataflow.memoization INFO: Task 259 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 260
parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/d2534519-81b6-462f-b5b4-4d7388e26bf9_isrForCrosstalkSources_855810_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/d2534519-81b6-462f-b5b4-4d7388e26bf9_isrForCrosstalkSources_855810_6.stderr
parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 260 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: <AppFuture at 0x1544622a3a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 260 has memoization hash 7a46106ab8eba84f32e38ffc2f87649d
parsl.dataflow.memoization INFO: Task 260 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 261
parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/868cbe03-9019-4c8d-980a-4071c2df3125_isrForCrosstalkSources_855810_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/868cbe03-9019-4c8d-980a-4071c2df3125_isrForCrosstalkSources_855810_43.stderr
parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 261 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: <AppFuture at 0x1544622a3e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 261 has memoization hash a9a2d1ca1feaca53d5cfdeac35373b71
parsl.dataflow.memoization INFO: Task 261 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 262
parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/3135be6f-15a6-4ff2-9a47-4bbd303ebe14_isrForCrosstalkSources_855717_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/3135be6f-15a6-4ff2-9a47-4bbd303ebe14_isrForCrosstalkSources_855717_10.stderr
parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 262 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: <AppFuture at 0x1544622a3610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 262 has memoization hash f8ca1035f81685008c1d569a2bb2a6c4
parsl.dataflow.memoization INFO: Task 262 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 263
parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/4227c6da-cd9c-4898-bf8f-ec1564b250ef_isrForCrosstalkSources_855808_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/4227c6da-cd9c-4898-bf8f-ec1564b250ef_isrForCrosstalkSources_855808_14.stderr
parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 263 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: <AppFuture at 0x154462298690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 263 has memoization hash ca162c174ddb8c2f23124d30cc4354c0
parsl.dataflow.memoization INFO: Task 263 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15446221c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 264
parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/dbae736c-692d-48d8-bdc5-0a5cb32a5eca_isrForCrosstalkSources_855810_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/dbae736c-692d-48d8-bdc5-0a5cb32a5eca_isrForCrosstalkSources_855810_7.stderr
parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 264 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: <AppFuture at 0x1544622a8050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 264 has memoization hash 27ed3d6d06696e5656949f784f8a5852
parsl.dataflow.memoization INFO: Task 264 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ac040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 265
parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/4e768d29-bb3f-4553-be41-208a692c1318_isrForCrosstalkSources_855718_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/4e768d29-bb3f-4553-be41-208a692c1318_isrForCrosstalkSources_855718_21.stderr
parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 265 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: <AppFuture at 0x1544622a9750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 58d09e291c0ea5858284f98ac02aa84c
parsl.dataflow.memoization INFO: Task 265 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ac180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 266
parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/6e983234-78dc-48fa-92e3-8d2904318dda_isrForCrosstalkSources_855716_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/6e983234-78dc-48fa-92e3-8d2904318dda_isrForCrosstalkSources_855716_58.stderr
parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 266 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: <AppFuture at 0x1544622aac90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 266 has memoization hash 7d4f2c2e40166440372fb59ebcd48b34
parsl.dataflow.memoization INFO: Task 266 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ac220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 267
parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/0becd143-4021-46a4-92ff-6da1703328a6_isrForCrosstalkSources_855718_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/0becd143-4021-46a4-92ff-6da1703328a6_isrForCrosstalkSources_855718_28.stderr
parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 267 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: <AppFuture at 0x1544622aa3d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 267 has memoization hash 03a8032024819fdbc9b7fef999b560c5
parsl.dataflow.memoization INFO: Task 267 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ac2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 268
parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/1ab74b15-38e0-48bc-a90c-8424730a17f0_isrForCrosstalkSources_855716_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/1ab74b15-38e0-48bc-a90c-8424730a17f0_isrForCrosstalkSources_855716_13.stderr
parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 268 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: <AppFuture at 0x1544622ab2d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 268 has memoization hash ef89a811d8a37aca34c729e3604f125f
parsl.dataflow.memoization INFO: Task 268 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ac400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 269
parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/e967644d-bf8d-4223-a761-a521d7c40c89_isrForCrosstalkSources_855716_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/e967644d-bf8d-4223-a761-a521d7c40c89_isrForCrosstalkSources_855716_25.stderr
parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 269 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: <AppFuture at 0x1544622aa710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 269 has memoization hash 8f73d38401decd94f1ec55d8da6f2127
parsl.dataflow.memoization INFO: Task 269 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ac360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 270
parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/117b2faa-cebf-46cd-9fd3-602ad5235784_isrForCrosstalkSources_855808_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/117b2faa-cebf-46cd-9fd3-602ad5235784_isrForCrosstalkSources_855808_33.stderr
parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 270 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: <AppFuture at 0x15446222cc90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 270 has memoization hash f8777c80893c9bba6d48dbf7d48a24c8
parsl.dataflow.memoization INFO: Task 270 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ac4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 271
parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/f7add256-d6d4-462c-97c1-647c7d178c55_isrForCrosstalkSources_855808_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/f7add256-d6d4-462c-97c1-647c7d178c55_isrForCrosstalkSources_855808_29.stderr
parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 271 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: <AppFuture at 0x1544622b5050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 271 has memoization hash e6dfb9d4a586c59a8ae5555e20bdbfae
parsl.dataflow.memoization INFO: Task 271 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ac540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 272
parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/c539ea85-1fe7-43a4-9703-4cfe7f3b60a3_isrForCrosstalkSources_855808_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/c539ea85-1fe7-43a4-9703-4cfe7f3b60a3_isrForCrosstalkSources_855808_54.stderr
parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 272 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: <AppFuture at 0x1544622b5250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 272 has memoization hash 6911b57bb7e575f7dff4dd136c42d78f
parsl.dataflow.memoization INFO: Task 272 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ac5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 273
parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/a439fccd-75d0-4f74-99fa-624672438a2d_isrForCrosstalkSources_855718_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/a439fccd-75d0-4f74-99fa-624672438a2d_isrForCrosstalkSources_855718_39.stderr
parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 273 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: <AppFuture at 0x1544622b60d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 273 has memoization hash d37c10faa0b6c0451b6dcbf18d1669cf
parsl.dataflow.memoization INFO: Task 273 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ac720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 274
parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/c6c11a86-657e-4ab0-8992-f3bd0dfb4045_isrForCrosstalkSources_855718_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/c6c11a86-657e-4ab0-8992-f3bd0dfb4045_isrForCrosstalkSources_855718_1.stderr
parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 274 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: <AppFuture at 0x1544622b6d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 5302c60787d64d54e572e082aed3ce27
parsl.dataflow.memoization INFO: Task 274 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ac680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 275
parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/b465fb30-fe7e-4490-9d9d-0f9f4a51414c_isrForCrosstalkSources_855809_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/b465fb30-fe7e-4490-9d9d-0f9f4a51414c_isrForCrosstalkSources_855809_58.stderr
parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 275 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: <AppFuture at 0x1544622b74d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 9271690015a42d53ae04b041a4fa3812
parsl.dataflow.memoization INFO: Task 275 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ac7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 276
parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/c749752a-c3ab-4631-92ed-b86cc0635446_isrForCrosstalkSources_855809_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/c749752a-c3ab-4631-92ed-b86cc0635446_isrForCrosstalkSources_855809_19.stderr
parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 276 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: <AppFuture at 0x1544622b7810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 276 has memoization hash cb43c32838e2452a55d0ac1e258d59c3
parsl.dataflow.memoization INFO: Task 276 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ac860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 277
parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/60ffea92-d07b-4a7f-a41c-a509c2fd31f1_isrForCrosstalkSources_855808_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/60ffea92-d07b-4a7f-a41c-a509c2fd31f1_isrForCrosstalkSources_855808_53.stderr
parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 277 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: <AppFuture at 0x1544622b7350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 277 has memoization hash 10cc73149eb125d2ef74a977e19ece69
parsl.dataflow.memoization INFO: Task 277 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ac900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 278
parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/f1ebf6ab-3ee6-4d48-8211-f5c4b4bb6b8f_isrForCrosstalkSources_855810_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/f1ebf6ab-3ee6-4d48-8211-f5c4b4bb6b8f_isrForCrosstalkSources_855810_52.stderr
parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 278 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: <AppFuture at 0x1544622b7d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 278 has memoization hash bac429c4b44718b159b96f653d222365
parsl.dataflow.memoization INFO: Task 278 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622aca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 279
parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/066fc771-8b2f-423a-9040-080a7b358439_isrForCrosstalkSources_855809_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/066fc771-8b2f-423a-9040-080a7b358439_isrForCrosstalkSources_855809_5.stderr
parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 279 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: <AppFuture at 0x1544622c4590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 279 has memoization hash d9e70aebb933a8751997c06c6c24d460
parsl.dataflow.memoization INFO: Task 279 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ac9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 280
parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/b18c6f0b-c71c-4ba5-a257-fca9251ab8ba_isrForCrosstalkSources_855716_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/b18c6f0b-c71c-4ba5-a257-fca9251ab8ba_isrForCrosstalkSources_855716_31.stderr
parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 280 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: <AppFuture at 0x1544622c4d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 280 has memoization hash b8134fb82c3f187795e89e5cc225a0ce
parsl.dataflow.memoization INFO: Task 280 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622acae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 281
parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/c6631729-423b-4c67-824d-0d8fb757aaaa_isrForCrosstalkSources_855717_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/c6631729-423b-4c67-824d-0d8fb757aaaa_isrForCrosstalkSources_855717_12.stderr
parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 281 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: <AppFuture at 0x1544622c55d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 281 has memoization hash 5efeb48e424f6bb67f46ce38816fc6ca
parsl.dataflow.memoization INFO: Task 281 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622acb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 282
parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/9c110c98-5a5e-4eb2-864b-03026bf6f05f_isrForCrosstalkSources_855808_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/9c110c98-5a5e-4eb2-864b-03026bf6f05f_isrForCrosstalkSources_855808_43.stderr
parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 282 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: <AppFuture at 0x1544622c6a50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 282 has memoization hash 6ab1a51e968639b0e70f4bed2a173105
parsl.dataflow.memoization INFO: Task 282 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622acc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 283
parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/50a17c2c-677b-469b-b664-57966ae1c92a_isrForCrosstalkSources_855718_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/50a17c2c-677b-469b-b664-57966ae1c92a_isrForCrosstalkSources_855718_29.stderr
parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 283 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: <AppFuture at 0x1544622c72d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 283 has memoization hash 14345ef658e0c93d784552a7cc243551
parsl.dataflow.memoization INFO: Task 283 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622acd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 284
parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/7be9c78f-9017-4a32-947b-9397ab108c70_isrForCrosstalkSources_855717_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/7be9c78f-9017-4a32-947b-9397ab108c70_isrForCrosstalkSources_855717_30.stderr
parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 284 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: <AppFuture at 0x1544622c71d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 284 has memoization hash 5681d94461fe94aa2f1842173095817b
parsl.dataflow.memoization INFO: Task 284 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622accc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 285
parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/8386d33a-5f36-4771-a593-76b7c5395275_isrForCrosstalkSources_855717_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/8386d33a-5f36-4771-a593-76b7c5395275_isrForCrosstalkSources_855717_35.stderr
parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 285 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: <AppFuture at 0x1544622a0850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 285 has memoization hash 46d169fbf2d9ed0cae268dfd360d7e53
parsl.dataflow.memoization INFO: Task 285 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ace00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 286
parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/61c76934-b461-4d8c-8c59-078a59de99b2_isrForCrosstalkSources_855810_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/61c76934-b461-4d8c-8c59-078a59de99b2_isrForCrosstalkSources_855810_32.stderr
parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 286 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: <AppFuture at 0x1544622c84d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 286 has memoization hash 5c7443de8d1077a3ab46db6a88015645
parsl.dataflow.memoization INFO: Task 286 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622acea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 287
parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/4b001e75-100b-4a1e-aa39-73ad2cccad2a_isrForCrosstalkSources_855810_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/4b001e75-100b-4a1e-aa39-73ad2cccad2a_isrForCrosstalkSources_855810_38.stderr
parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 287 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: <AppFuture at 0x1544622c9710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 9fb08a471d1d0586d613bb36fa1bf6d3
parsl.dataflow.memoization INFO: Task 287 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622acf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 288
parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/d6f96736-f9f0-4925-8fb3-97ef2983a4d4_isrForCrosstalkSources_855716_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/d6f96736-f9f0-4925-8fb3-97ef2983a4d4_isrForCrosstalkSources_855716_50.stderr
parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 288 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: <AppFuture at 0x1544622ca950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 288 has memoization hash cfd6f4cce09acc9e0300de8e4102cd6d
parsl.dataflow.memoization INFO: Task 288 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ad080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 289
parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/d773fed9-d4aa-4caa-955c-e477eb17b0b0_isrForCrosstalkSources_855716_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/d773fed9-d4aa-4caa-955c-e477eb17b0b0_isrForCrosstalkSources_855716_20.stderr
parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 289 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: <AppFuture at 0x1544622cb1d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 53951ad58ee7df4b34f1ef41c04d51ef
parsl.dataflow.memoization INFO: Task 289 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622acfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 290
parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/868ef362-f9ae-442c-bb94-8a99c7c4c4bd_isrForCrosstalkSources_855717_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/868ef362-f9ae-442c-bb94-8a99c7c4c4bd_isrForCrosstalkSources_855717_5.stderr
parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 290 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: <AppFuture at 0x1544622cac90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 290 has memoization hash 3f9cc38c61f58417616ea12a48a83076
parsl.dataflow.memoization INFO: Task 290 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ad120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 291
parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/98cf0e9f-c333-4e29-8c19-6347dd316cdd_isrForCrosstalkSources_855718_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/98cf0e9f-c333-4e29-8c19-6347dd316cdd_isrForCrosstalkSources_855718_24.stderr
parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 291 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: <AppFuture at 0x1544622cb550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 291 has memoization hash b172c9c9d86f11e2ec78312d5251914d
parsl.dataflow.memoization INFO: Task 291 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ad1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 292
parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/4112f4d4-0b7a-480c-bcb2-9a3dbff375df_isrForCrosstalkSources_855808_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/4112f4d4-0b7a-480c-bcb2-9a3dbff375df_isrForCrosstalkSources_855808_30.stderr
parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 292 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: <AppFuture at 0x1544622cbb10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 292 has memoization hash 43b74b0242bb6c62bc8c925fb37e6028
parsl.dataflow.memoization INFO: Task 292 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ad260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 293
parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/bd578b3a-3eb4-4c6f-8eb3-f7d9a4d992b7_isrForCrosstalkSources_855718_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/bd578b3a-3eb4-4c6f-8eb3-f7d9a4d992b7_isrForCrosstalkSources_855718_42.stderr
parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 293 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: <AppFuture at 0x1544622d8310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 293 has memoization hash 5dd6bf37cec3cc16f305213f491a6a30
parsl.dataflow.memoization INFO: Task 293 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ad3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 294
parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/fe94f797-3dc1-4e0d-b175-785e7a65adbb_isrForCrosstalkSources_855808_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/fe94f797-3dc1-4e0d-b175-785e7a65adbb_isrForCrosstalkSources_855808_60.stderr
parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 294 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: <AppFuture at 0x1544622d8f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 294 has memoization hash 7725e53ca0a7f5c52e57aa84d26896fe
parsl.dataflow.memoization INFO: Task 294 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ad300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 295
parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/970352a1-db6f-4743-8e61-f2a6320f5427_isrForCrosstalkSources_855808_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/970352a1-db6f-4743-8e61-f2a6320f5427_isrForCrosstalkSources_855808_47.stderr
parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 295 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: <AppFuture at 0x1544622d9610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 295 has memoization hash c7068f0924c5bc2614a93193457c822e
parsl.dataflow.memoization INFO: Task 295 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ad440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 296
parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/1bd4a503-42f4-48b2-94c7-f31be150e0cf_isrForCrosstalkSources_855810_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/1bd4a503-42f4-48b2-94c7-f31be150e0cf_isrForCrosstalkSources_855810_47.stderr
parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 296 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: <AppFuture at 0x1544622da350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 99ecfb29ff01b5cba1bb06079ae42f4d
parsl.dataflow.memoization INFO: Task 296 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ad4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 297
parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/dacfdb29-42b4-4e0d-884f-31a670bfa5cb_isrForCrosstalkSources_855810_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/dacfdb29-42b4-4e0d-884f-31a670bfa5cb_isrForCrosstalkSources_855810_19.stderr
parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 297 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: <AppFuture at 0x1544622db6d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 8f8a1ad88489369dff8628bfc1631886
parsl.dataflow.memoization INFO: Task 297 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ad580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 298
parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/66160643-0138-49e2-8ce3-cc3efb24874b_isrForCrosstalkSources_855809_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/66160643-0138-49e2-8ce3-cc3efb24874b_isrForCrosstalkSources_855809_49.stderr
parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 298 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: <AppFuture at 0x1544622dae10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 298 has memoization hash 5ea411fafa84f1d68f891ee4ff41ef86
parsl.dataflow.memoization INFO: Task 298 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ad6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 299
parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/5eeb69a9-688e-4fb6-a71c-71d50bf318b9_isrForCrosstalkSources_855810_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/5eeb69a9-688e-4fb6-a71c-71d50bf318b9_isrForCrosstalkSources_855810_33.stderr
parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 299 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: <AppFuture at 0x1544622db310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 299 has memoization hash f2a31cb5d1dd780703a31be6beacf52d
parsl.dataflow.memoization INFO: Task 299 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ad620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 300
parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/4901c0ad-80ac-407a-8816-c58c4f501aa7_isrForCrosstalkSources_855716_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/4901c0ad-80ac-407a-8816-c58c4f501aa7_isrForCrosstalkSources_855716_47.stderr
parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 300 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: <AppFuture at 0x1544622c44d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 300 has memoization hash 8a84a7f7f9a7192634bee25e20a5278e
parsl.dataflow.memoization INFO: Task 300 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ad760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 301
parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/5e638a43-2528-4e0f-853b-1d508737cdfb_isrForCrosstalkSources_855810_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/5e638a43-2528-4e0f-853b-1d508737cdfb_isrForCrosstalkSources_855810_55.stderr
parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 301 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: <AppFuture at 0x1544622e4e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 301 has memoization hash 19cc75c7bd87f23d0697ebae4a29908d
parsl.dataflow.memoization INFO: Task 301 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ad800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 302
parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/bf6612c4-94b8-4d1e-b382-d921dd09a56e_isrForCrosstalkSources_855808_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/bf6612c4-94b8-4d1e-b382-d921dd09a56e_isrForCrosstalkSources_855808_3.stderr
parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 302 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: <AppFuture at 0x15446224e210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 302 has memoization hash 38287291ee99bc0a4e06f5088129bcab
parsl.dataflow.memoization INFO: Task 302 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ad8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 303
parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/4dc23d5b-78b4-476b-9d05-a35a436d7b2d_isrForCrosstalkSources_855716_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/4dc23d5b-78b4-476b-9d05-a35a436d7b2d_isrForCrosstalkSources_855716_60.stderr
parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 303 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: <AppFuture at 0x1544622e5d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 303 has memoization hash 9eb9c72536284b51eebf769032809bd2
parsl.dataflow.memoization INFO: Task 303 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ad9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 304
parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/5213aaa5-fdc0-4bff-baa7-c799501c3e9a_isrForCrosstalkSources_855809_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/5213aaa5-fdc0-4bff-baa7-c799501c3e9a_isrForCrosstalkSources_855809_51.stderr
parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 304 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: <AppFuture at 0x1544622e5a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 304 has memoization hash ca8f64c67dfae9fccfef6d939a00cb5e
parsl.dataflow.memoization INFO: Task 304 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ad940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 305
parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/c285269f-0b30-4aa7-aec8-470da365913f_isrForCrosstalkSources_855716_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/c285269f-0b30-4aa7-aec8-470da365913f_isrForCrosstalkSources_855716_15.stderr
parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 305 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: <AppFuture at 0x1544622e6dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 305 has memoization hash e889df5d4450569ac41eb3e8df98710d
parsl.dataflow.memoization INFO: Task 305 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ada80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 306
parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/61f39293-5225-44ca-8cd1-5790c1017042_isrForCrosstalkSources_855717_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/61f39293-5225-44ca-8cd1-5790c1017042_isrForCrosstalkSources_855717_29.stderr
parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 306 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: <AppFuture at 0x1544622e7610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 306 has memoization hash ff260bb047ed2981421d1cc434e17fcd
parsl.dataflow.memoization INFO: Task 306 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622adb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 307
parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/3d4c4cee-768e-4a78-be5f-5cf320462ae4_isrForCrosstalkSources_855717_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/3d4c4cee-768e-4a78-be5f-5cf320462ae4_isrForCrosstalkSources_855717_7.stderr
parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 307 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: <AppFuture at 0x1544622e7e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 307 has memoization hash 4c7b4de85f8855cf204fe3a71bc05803
parsl.dataflow.memoization INFO: Task 307 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622adbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 308
parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/d686ee17-5bcd-4fef-b810-6c77d4d42f06_isrForCrosstalkSources_855810_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/d686ee17-5bcd-4fef-b810-6c77d4d42f06_isrForCrosstalkSources_855810_46.stderr
parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 308 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: <AppFuture at 0x1544622c8650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 308 has memoization hash 84965e5be1c9d29c70824b335db67145
parsl.dataflow.memoization INFO: Task 308 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622add00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 309
parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/8d2eacf2-7d88-45af-b5db-a401e174811c_isrForCrosstalkSources_855716_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/8d2eacf2-7d88-45af-b5db-a401e174811c_isrForCrosstalkSources_855716_21.stderr
parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 309 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: <AppFuture at 0x1544620f11d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 94abd6b58c81e3b6a01da4672fe23a0e
parsl.dataflow.memoization INFO: Task 309 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622adc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 310
parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/5ceeac75-cf18-469f-9044-5464be0f71bb_isrForCrosstalkSources_855810_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/5ceeac75-cf18-469f-9044-5464be0f71bb_isrForCrosstalkSources_855810_26.stderr
parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 310 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: <AppFuture at 0x1544620f1710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 25aaf96f8ec658e604acb25d2b61a8c0
parsl.dataflow.memoization INFO: Task 310 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622adda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 311
parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/f825faf8-41e3-4e86-8c47-70a8ba97aa5c_isrForCrosstalkSources_855810_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/f825faf8-41e3-4e86-8c47-70a8ba97aa5c_isrForCrosstalkSources_855810_23.stderr
parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 311 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: <AppFuture at 0x1544620f2010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 311 has memoization hash 36d4c6dd58b89ff0b3a6d8b079ba117b
parsl.dataflow.memoization INFO: Task 311 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ade40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 312
parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/80dcbaf1-f4af-48e9-811d-dd105416a76c_isrForCrosstalkSources_855718_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/80dcbaf1-f4af-48e9-811d-dd105416a76c_isrForCrosstalkSources_855718_18.stderr
parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 312 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: <AppFuture at 0x1544620f2a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 1d705f4a103d7d15713e1c17efa4c624
parsl.dataflow.memoization INFO: Task 312 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622adee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 313
parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/77fb20a2-0348-4129-b70b-e43f4fe3556e_isrForCrosstalkSources_855809_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/77fb20a2-0348-4129-b70b-e43f4fe3556e_isrForCrosstalkSources_855809_56.stderr
parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 313 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: <AppFuture at 0x1544620f38d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 313 has memoization hash 3cd6f4c202f318d769f0781d7cab2348
parsl.dataflow.memoization INFO: Task 313 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ae020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 314
parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/0c576e78-b4e0-4749-a776-7d83e4e4048c_isrForCrosstalkSources_855716_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/0c576e78-b4e0-4749-a776-7d83e4e4048c_isrForCrosstalkSources_855716_17.stderr
parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 314 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: <AppFuture at 0x1544620f2ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 314 has memoization hash f12e8ab95845731e2cf435e47d096f30
parsl.dataflow.memoization INFO: Task 314 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622adf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 315
parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/3d43dd92-26dd-449a-b077-e3f7934491fb_isrForCrosstalkSources_855808_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/3d43dd92-26dd-449a-b077-e3f7934491fb_isrForCrosstalkSources_855808_40.stderr
parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 315 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: <AppFuture at 0x1544620f3f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 315 has memoization hash cb7fb2789b57c62b66c3ee43cb657275
parsl.dataflow.memoization INFO: Task 315 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ae0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 316
parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/37928a4d-13b4-4ef5-a590-7a6bd93d072c_isrForCrosstalkSources_855810_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/37928a4d-13b4-4ef5-a590-7a6bd93d072c_isrForCrosstalkSources_855810_42.stderr
parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 316 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: <AppFuture at 0x1544620f8910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 88ae57b4398d8369b1c84f437fb4350f
parsl.dataflow.memoization INFO: Task 316 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ae160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 317
parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/0df094f8-7324-4267-89b5-2ce30afa6bb2_isrForCrosstalkSources_855718_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/0df094f8-7324-4267-89b5-2ce30afa6bb2_isrForCrosstalkSources_855718_41.stderr
parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 317 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: <AppFuture at 0x1544620fa110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 317 has memoization hash 01b497d10ca9aa461c96aa79c6b5b768
parsl.dataflow.memoization INFO: Task 317 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ae200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 318
parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/d13abbd0-18c5-4efb-b592-32fa1798eee7_isrForCrosstalkSources_855716_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/d13abbd0-18c5-4efb-b592-32fa1798eee7_isrForCrosstalkSources_855716_52.stderr
parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 318 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: <AppFuture at 0x1544620fa250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 318 has memoization hash 03eb941eed50e0b00c0877641f3284b1
parsl.dataflow.memoization INFO: Task 318 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ae340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 319
parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/365e7f8e-4159-40d2-aa06-81cc6f0ab058_isrForCrosstalkSources_855810_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/365e7f8e-4159-40d2-aa06-81cc6f0ab058_isrForCrosstalkSources_855810_34.stderr
parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 319 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: <AppFuture at 0x1544620fa750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 1c79b372f8de7683b675fe0892c94153
parsl.dataflow.memoization INFO: Task 319 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ae2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 320
parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/908b3ba5-0565-4ae1-9898-a427de84ffce_isrForCrosstalkSources_855716_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/908b3ba5-0565-4ae1-9898-a427de84ffce_isrForCrosstalkSources_855716_7.stderr
parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 320 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: <AppFuture at 0x1544620fac90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 320 has memoization hash ced53466299ed6492c328bf427b89f76
parsl.dataflow.memoization INFO: Task 320 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ae3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 321
parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/359fdaea-3b89-4dcc-9485-01588fa41891_isrForCrosstalkSources_855717_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/359fdaea-3b89-4dcc-9485-01588fa41891_isrForCrosstalkSources_855717_11.stderr
parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 321 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: <AppFuture at 0x1544620fbf90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 321 has memoization hash 8edcc05786b56a6ba1eb39cd78a54048
parsl.dataflow.memoization INFO: Task 321 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ae480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 322
parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/182cef8e-735d-4230-90c6-191b8b7f7d7a_isrForCrosstalkSources_855809_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/182cef8e-735d-4230-90c6-191b8b7f7d7a_isrForCrosstalkSources_855809_57.stderr
parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 322 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: <AppFuture at 0x1544620fb2d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 322 has memoization hash 92133535b28850cdfc3dc58bdc10e025
parsl.dataflow.memoization INFO: Task 322 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ae520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 323
parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/227f0264-3fda-4e3d-b00b-f0fac3a7b510_isrForCrosstalkSources_855716_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/227f0264-3fda-4e3d-b00b-f0fac3a7b510_isrForCrosstalkSources_855716_1.stderr
parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 323 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: <AppFuture at 0x1544622e4c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 323 has memoization hash b1e3b01cd17977ff84ea92bb64a5e58b
parsl.dataflow.memoization INFO: Task 323 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ae660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 324
parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/7a082fa1-9173-4249-a726-12bab62a6c9d_isrForCrosstalkSources_855716_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/7a082fa1-9173-4249-a726-12bab62a6c9d_isrForCrosstalkSources_855716_32.stderr
parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 324 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: <AppFuture at 0x1544620fd3d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 324 has memoization hash a030b89ce434b7db60ab72dcfd75d9fd
parsl.dataflow.memoization INFO: Task 324 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ae5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 325
parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/d4d7f8ca-9750-4ca9-9a31-1444b0897186_isrForCrosstalkSources_855717_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/d4d7f8ca-9750-4ca9-9a31-1444b0897186_isrForCrosstalkSources_855717_61.stderr
parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 325 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: <AppFuture at 0x1544620fda50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 216e36229b5f317354937290483e3ceb
parsl.dataflow.memoization INFO: Task 325 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ae700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 326
parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/c0524524-11aa-4621-adf6-4db829380ddc_isrForCrosstalkSources_855716_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/c0524524-11aa-4621-adf6-4db829380ddc_isrForCrosstalkSources_855716_45.stderr
parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 326 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: <AppFuture at 0x1544620fd890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 326 has memoization hash 8cbae8c56753c481ce57e537c51e4307
parsl.dataflow.memoization INFO: Task 326 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ae7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 327
parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/54a8b699-6024-447c-9425-c49e932d6b2e_isrForCrosstalkSources_855808_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/54a8b699-6024-447c-9425-c49e932d6b2e_isrForCrosstalkSources_855808_12.stderr
parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 327 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: <AppFuture at 0x1544620fe550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 327 has memoization hash c19d65c0d35bcf3cfcc91090fb5e977f
parsl.dataflow.memoization INFO: Task 327 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ae840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 328
parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/9fe358c8-5f59-4bdf-9abb-9096c9bff808_isrForCrosstalkSources_855809_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/9fe358c8-5f59-4bdf-9abb-9096c9bff808_isrForCrosstalkSources_855809_28.stderr
parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 328 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: <AppFuture at 0x1544620ffb90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 328 has memoization hash 0cab8ccacc3380fea1db13311a1ed5aa
parsl.dataflow.memoization INFO: Task 328 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ae980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 329
parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/7d18c17b-c097-4f3f-bd7e-045c573fb50f_isrForCrosstalkSources_855717_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/7d18c17b-c097-4f3f-bd7e-045c573fb50f_isrForCrosstalkSources_855717_8.stderr
parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 329 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: <AppFuture at 0x1544620ff010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 329 has memoization hash 23a9dc1f2503ddc22204b6df2fe1abd4
parsl.dataflow.memoization INFO: Task 329 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ae8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 330
parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/54c7c32a-b1af-4005-a831-87e7c65c6ef8_isrForCrosstalkSources_855716_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/54c7c32a-b1af-4005-a831-87e7c65c6ef8_isrForCrosstalkSources_855716_48.stderr
parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 330 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: <AppFuture at 0x1544620f0f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 59a816459a4ed6dee1e6c96f41c96821
parsl.dataflow.memoization INFO: Task 330 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622aea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 331
parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/0af08179-9689-47bc-ad9e-b0a238bc6156_isrForCrosstalkSources_855809_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/0af08179-9689-47bc-ad9e-b0a238bc6156_isrForCrosstalkSources_855809_55.stderr
parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 331 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: <AppFuture at 0x1544620fee50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 765dbad89c33b35993e25db1c4986696
parsl.dataflow.memoization INFO: Task 331 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622aeac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 332
parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/98ed8f42-fb13-4e29-a242-bf6ef2d195d1_isrForCrosstalkSources_855718_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/98ed8f42-fb13-4e29-a242-bf6ef2d195d1_isrForCrosstalkSources_855718_35.stderr
parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 332 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: <AppFuture at 0x1544622c9210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 332 has memoization hash b76153123fb2f3952af9836bd24edd02
parsl.dataflow.memoization INFO: Task 332 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622aeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 333
parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/8178416b-6755-4d7a-ae41-119b57707b54_isrForCrosstalkSources_855717_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/8178416b-6755-4d7a-ae41-119b57707b54_isrForCrosstalkSources_855717_21.stderr
parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 333 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: <AppFuture at 0x154462119b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 333 has memoization hash 04e6b4b5544a30321a5cef938708ae64
parsl.dataflow.memoization INFO: Task 333 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622aeca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 334
parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/cd7a2ecd-26f5-4c8d-b35e-b1bde14f62ab_isrForCrosstalkSources_855810_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/cd7a2ecd-26f5-4c8d-b35e-b1bde14f62ab_isrForCrosstalkSources_855810_56.stderr
parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 334 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: <AppFuture at 0x15446211a050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 68fbb605c571f9161fa6e7fd3ba4dc64
parsl.dataflow.memoization INFO: Task 334 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622aec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 335
parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/a5e030fd-6c5a-409a-a1a5-aecda345aa4d_isrForCrosstalkSources_855808_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/a5e030fd-6c5a-409a-a1a5-aecda345aa4d_isrForCrosstalkSources_855808_44.stderr
parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 335 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: <AppFuture at 0x15446211acd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 335 has memoization hash 975e9441dc93d3db44bf5b569f89c951
parsl.dataflow.memoization INFO: Task 335 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622aed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 336
parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/77523463-4689-4989-ae02-4116d0237ae4_isrForCrosstalkSources_855808_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/77523463-4689-4989-ae02-4116d0237ae4_isrForCrosstalkSources_855808_1.stderr
parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 336 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: <AppFuture at 0x15446211b0d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 336 has memoization hash cef8ce6e2726bdc2933da2b372202733
parsl.dataflow.memoization INFO: Task 336 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622aede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 337
parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/0a1b3ab7-9bb5-4cc5-8e04-46e1d3e5f122_isrForCrosstalkSources_855716_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/0a1b3ab7-9bb5-4cc5-8e04-46e1d3e5f122_isrForCrosstalkSources_855716_36.stderr
parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 337 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: <AppFuture at 0x15446211ad10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 337 has memoization hash 450827e055b3bad8812220cd0e183761
parsl.dataflow.memoization INFO: Task 337 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622aee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 338
parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/1c344fc0-d9d8-4266-a857-5dbfd05a9c13_isrForCrosstalkSources_855718_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/1c344fc0-d9d8-4266-a857-5dbfd05a9c13_isrForCrosstalkSources_855718_51.stderr
parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 338 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: <AppFuture at 0x154462120710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 338 has memoization hash 7cd970cd0f6c298c674d905b9cdc56fa
parsl.dataflow.memoization INFO: Task 338 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622aefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 339
parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/afa2ecb7-2d4e-4472-9cf6-4d26481ee39c_isrForCrosstalkSources_855717_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/afa2ecb7-2d4e-4472-9cf6-4d26481ee39c_isrForCrosstalkSources_855717_45.stderr
parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 339 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: <AppFuture at 0x1544622d8f10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 339 has memoization hash a3bac18e1b49dd38fcecbf283998b98a
parsl.dataflow.memoization INFO: Task 339 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622aef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 340
parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/8961446a-2da1-4d67-9d69-db456295ac01_isrForCrosstalkSources_855718_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/8961446a-2da1-4d67-9d69-db456295ac01_isrForCrosstalkSources_855718_5.stderr
parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 340 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: <AppFuture at 0x154462122010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 340 has memoization hash 0fb31f04fb8028abfbfd4a51b16cc153
parsl.dataflow.memoization INFO: Task 340 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622af060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 341
parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/a69a3a20-2f63-4b19-9b2d-e4293df9c66a_isrForCrosstalkSources_855717_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/a69a3a20-2f63-4b19-9b2d-e4293df9c66a_isrForCrosstalkSources_855717_49.stderr
parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 341 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: <AppFuture at 0x154462121f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 341 has memoization hash 996e926cd5bc6f6dcc080d2803b255bc
parsl.dataflow.memoization INFO: Task 341 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622af100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 342
parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/b3487d17-c7ed-4abf-820e-47da32589139_isrForCrosstalkSources_855808_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/b3487d17-c7ed-4abf-820e-47da32589139_isrForCrosstalkSources_855808_26.stderr
parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 342 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: <AppFuture at 0x154462122890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 342 has memoization hash f6fec0a91805f8b0253544ad83faa0b0
parsl.dataflow.memoization INFO: Task 342 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622af1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 343
parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/af156947-93ce-4c74-a210-22d3a49dbb91_isrForCrosstalkSources_855717_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/af156947-93ce-4c74-a210-22d3a49dbb91_isrForCrosstalkSources_855717_52.stderr
parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 343 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: <AppFuture at 0x154462123210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 343 has memoization hash aa6a627513455ba05624fb693c3bac9d
parsl.dataflow.memoization INFO: Task 343 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622af2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 344
parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/692ecc34-a6ff-40cd-9eb9-25a2b86560df_isrForCrosstalkSources_855808_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/692ecc34-a6ff-40cd-9eb9-25a2b86560df_isrForCrosstalkSources_855808_31.stderr
parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 344 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: <AppFuture at 0x154462120bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 344 has memoization hash cbd3b764b6612de31ca7d5f1bcbde62f
parsl.dataflow.memoization INFO: Task 344 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622af240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 345
parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/8c0ffa0f-4555-452f-a130-c29952cdedd0_isrForCrosstalkSources_855717_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/8c0ffa0f-4555-452f-a130-c29952cdedd0_isrForCrosstalkSources_855717_53.stderr
parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 345 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: <AppFuture at 0x1544622e4990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 345 has memoization hash f5a626b682509723edfc83fce582d1fe
parsl.dataflow.memoization INFO: Task 345 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622af380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 346
parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/0bf5ea5c-9a7a-4e5c-bfb6-e733d5016499_isrForCrosstalkSources_855808_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/0bf5ea5c-9a7a-4e5c-bfb6-e733d5016499_isrForCrosstalkSources_855808_7.stderr
parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 346 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: <AppFuture at 0x154462124d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 346 has memoization hash 33231b6a3340f764feef49f4afa4fd52
parsl.dataflow.memoization INFO: Task 346 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622af420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 347
parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/b59541fd-6803-4487-9318-5dcaed407b18_isrForCrosstalkSources_855809_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/b59541fd-6803-4487-9318-5dcaed407b18_isrForCrosstalkSources_855809_15.stderr
parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 347 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: <AppFuture at 0x154462125690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 347 has memoization hash c82f37dd9a3f1d6f8f45f646de7cbd0d
parsl.dataflow.memoization INFO: Task 347 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622af4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 348
parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/af097a2a-d64e-4bc2-95b3-07384d339145_isrForCrosstalkSources_855808_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/af097a2a-d64e-4bc2-95b3-07384d339145_isrForCrosstalkSources_855808_11.stderr
parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 348 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: <AppFuture at 0x154462125e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 348 has memoization hash 1c438c593b61fd11477e3013df3bb007
parsl.dataflow.memoization INFO: Task 348 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622af600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 349
parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/3e3138ea-ec2f-4b9a-b473-1c8025f86aa1_isrForCrosstalkSources_855810_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/3e3138ea-ec2f-4b9a-b473-1c8025f86aa1_isrForCrosstalkSources_855810_13.stderr
parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 349 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: <AppFuture at 0x154462126550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 349 has memoization hash b211f346692d969e719a53b1c3b3826f
parsl.dataflow.memoization INFO: Task 349 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622af560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 350
parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/b79c1360-3297-4509-bb61-b8e8d3f42712_isrForCrosstalkSources_855809_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/b79c1360-3297-4509-bb61-b8e8d3f42712_isrForCrosstalkSources_855809_39.stderr
parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 350 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: <AppFuture at 0x154462126cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 350 has memoization hash 436f1c74f0c18dc4f67f8a9a52395a27
parsl.dataflow.memoization INFO: Task 350 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622af6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 351
parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/82d11e33-0cd8-4657-af1e-8f440e8044af_isrForCrosstalkSources_855810_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/82d11e33-0cd8-4657-af1e-8f440e8044af_isrForCrosstalkSources_855810_41.stderr
parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 351 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: <AppFuture at 0x154462126b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 351 has memoization hash b0222557dc45768115f9c020e6629526
parsl.dataflow.memoization INFO: Task 351 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622af740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 352
parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/a3842a81-47f2-461a-b430-66ce0523769e_isrForCrosstalkSources_855717_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/a3842a81-47f2-461a-b430-66ce0523769e_isrForCrosstalkSources_855717_24.stderr
parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 352 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: <AppFuture at 0x154462127e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 8eb4280bde6fb05adeedb7a5a80aedd8
parsl.dataflow.memoization INFO: Task 352 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622af7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 353
parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/4d666a1f-0954-42ce-ac89-95969459291a_isrForCrosstalkSources_855718_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/4d666a1f-0954-42ce-ac89-95969459291a_isrForCrosstalkSources_855718_58.stderr
parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 353 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: <AppFuture at 0x154462119010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 353 has memoization hash 020153ab774a1be8d792ad2d7112bbd4
parsl.dataflow.memoization INFO: Task 353 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622af920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 354
parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/d6e4f6d8-2497-4df7-abb4-70e6d22aa7b4_isrForCrosstalkSources_855809_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/d6e4f6d8-2497-4df7-abb4-70e6d22aa7b4_isrForCrosstalkSources_855809_14.stderr
parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 354 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: <AppFuture at 0x154462130c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 354 has memoization hash 58da19817636469e0bb4856a1da16fe8
parsl.dataflow.memoization INFO: Task 354 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622af880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 355
parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/13b5fb80-8934-47a0-afbd-99d5e2bfa2b4_isrForCrosstalkSources_855716_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/13b5fb80-8934-47a0-afbd-99d5e2bfa2b4_isrForCrosstalkSources_855716_33.stderr
parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 355 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: <AppFuture at 0x154462131f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 355 has memoization hash f5e3f51049c9ae090ac6adc358970659
parsl.dataflow.memoization INFO: Task 355 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622af9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 356
parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/0ae64c8a-e299-4648-a30b-1cb5ec3c357f_isrForCrosstalkSources_855718_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/0ae64c8a-e299-4648-a30b-1cb5ec3c357f_isrForCrosstalkSources_855718_44.stderr
parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 356 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: <AppFuture at 0x154462133010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 356 has memoization hash 679d9bbff69d36a9347fd33804fc3ea1
parsl.dataflow.memoization INFO: Task 356 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622afa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 357
parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/c3d578d8-ae02-41a7-8cd5-83cb769e6a5a_isrForCrosstalkSources_855809_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/c3d578d8-ae02-41a7-8cd5-83cb769e6a5a_isrForCrosstalkSources_855809_38.stderr
parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 357 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: <AppFuture at 0x154462132a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 357 has memoization hash 41e80ddd0337d3d71c39deae0da4c3f6
parsl.dataflow.memoization INFO: Task 357 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622afb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 358
parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/8078361a-58cd-4e1e-8310-29baa4bc8eb6_isrForCrosstalkSources_855718_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/8078361a-58cd-4e1e-8310-29baa4bc8eb6_isrForCrosstalkSources_855718_54.stderr
parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 358 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: <AppFuture at 0x1544621336d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 358 has memoization hash 2277f8fa1625425da15c333d7000d953
parsl.dataflow.memoization INFO: Task 358 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622afc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 359
parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/1efa9402-4e53-4d5b-985c-388ae5f5b4e7_isrForCrosstalkSources_855809_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/1efa9402-4e53-4d5b-985c-388ae5f5b4e7_isrForCrosstalkSources_855809_10.stderr
parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 359 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: <AppFuture at 0x154462131f10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 359 has memoization hash fb50e1e84cc2aa7468c9b9500b1406c2
parsl.dataflow.memoization INFO: Task 359 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622afba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 360
parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/c31939ee-3aae-4e61-a4d2-35186c973490_isrForCrosstalkSources_855718_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/c31939ee-3aae-4e61-a4d2-35186c973490_isrForCrosstalkSources_855718_34.stderr
parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 360 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: <AppFuture at 0x154462132c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 360 has memoization hash 4aea1a27934ab381d50368d9312c942d
parsl.dataflow.memoization INFO: Task 360 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622afce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 361
parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/46c95bff-60e2-4165-b13b-52b8a0225c95_isrForCrosstalkSources_855810_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/46c95bff-60e2-4165-b13b-52b8a0225c95_isrForCrosstalkSources_855810_36.stderr
parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 361 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: <AppFuture at 0x15446213d050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 361 has memoization hash fdc42f7a0b7786c24d6cc3b04abde9a7
parsl.dataflow.memoization INFO: Task 361 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622afd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 362
parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/c2a5b9aa-6faa-494d-8637-9905a6f1bd3e_isrForCrosstalkSources_855717_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/c2a5b9aa-6faa-494d-8637-9905a6f1bd3e_isrForCrosstalkSources_855717_56.stderr
parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 362 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: <AppFuture at 0x15446213c890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 362 has memoization hash 55bc755c842d1a71ce00f8b6e9b5f0dd
parsl.dataflow.memoization INFO: Task 362 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622afe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 363
parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/e3eabb20-2264-48e8-8511-479d7c725865_isrForCrosstalkSources_855717_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/e3eabb20-2264-48e8-8511-479d7c725865_isrForCrosstalkSources_855717_9.stderr
parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 363 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: <AppFuture at 0x15446213d450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 363 has memoization hash e617ab8768fe9b74126c862651c66325
parsl.dataflow.memoization INFO: Task 363 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622aff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 364
parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/0d6ed138-aad3-421c-85f6-388d9520e6fb_isrForCrosstalkSources_855809_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/0d6ed138-aad3-421c-85f6-388d9520e6fb_isrForCrosstalkSources_855809_9.stderr
parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 364 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: <AppFuture at 0x15446213d810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 364 has memoization hash 13eac902a2200f02088cd30c62c89064
parsl.dataflow.memoization INFO: Task 364 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622afec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 365
parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/1ce2a30f-065e-4dc7-9880-f199c70511a3_isrForCrosstalkSources_855717_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/1ce2a30f-065e-4dc7-9880-f199c70511a3_isrForCrosstalkSources_855717_23.stderr
parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 365 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: <AppFuture at 0x15446213f090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 365 has memoization hash ed1c283d3f9f8cd204c0c488e045b39f
parsl.dataflow.memoization INFO: Task 365 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544622ac0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 366
parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/08b3b017-c3fe-448a-b228-d85a275c6394_isrForCrosstalkSources_855716_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/08b3b017-c3fe-448a-b228-d85a275c6394_isrForCrosstalkSources_855716_56.stderr
parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 366 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: <AppFuture at 0x15446213f610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 366 has memoization hash e94af7c5738243b26b43c6ff4578aa0e
parsl.dataflow.memoization INFO: Task 366 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544621400e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 367
parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/9d407bd9-c54c-433d-af28-e7382a7e6dd8_isrForCrosstalkSources_855718_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/9d407bd9-c54c-433d-af28-e7382a7e6dd8_isrForCrosstalkSources_855718_19.stderr
parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 367 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: <AppFuture at 0x15446213ee90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 367 has memoization hash 627dde1b6fbb05734bf50a6749778596
parsl.dataflow.memoization INFO: Task 367 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x154462140180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 368
parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/ffd0b4a8-c42b-4f99-a3aa-acc3b2fa3ae1_isrForCrosstalkSources_855808_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/ffd0b4a8-c42b-4f99-a3aa-acc3b2fa3ae1_isrForCrosstalkSources_855808_13.stderr
parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 368 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: <AppFuture at 0x1544621481d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 368 has memoization hash 471aa665b51f8242916ad027769aa59a
parsl.dataflow.memoization INFO: Task 368 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1544621402c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 369
parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/a15ce514-15f0-456f-a38d-e24ae0eb3c4b_isrForCrosstalkSources_855810_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/a15ce514-15f0-456f-a38d-e24ae0eb3c4b_isrForCrosstalkSources_855810_10.stderr
parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 369 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: <AppFuture at 0x154462148f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 369 has memoization hash db54efaf9a2de9c409de5f7a9ea74c0b
parsl.dataflow.memoization INFO: Task 369 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x154462140220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 370
parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/48ed2d7d-f6a6-405a-88f5-98b4b2d2550d_isrForCrosstalkSources_855809_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/48ed2d7d-f6a6-405a-88f5-98b4b2d2550d_isrForCrosstalkSources_855809_12.stderr
parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 370 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: <AppFuture at 0x154462149d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 370 has memoization hash 8154097830c9cfdd12c4783c6289d900
parsl.dataflow.memoization INFO: Task 370 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x154462140360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 371
parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/35d3ccbe-d38e-4103-b8ad-fe189d5577ce_isrForCrosstalkSources_855809_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/35d3ccbe-d38e-4103-b8ad-fe189d5577ce_isrForCrosstalkSources_855809_24.stderr
parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 371 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: <AppFuture at 0x15446214a150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 371 has memoization hash d13cbd5f6ff9f913eb71b3cd2dd82cce
parsl.dataflow.memoization INFO: Task 371 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x154462140400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 372
parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/7bdba7bc-7091-4bb8-947b-0abdc132ba19_isrForCrosstalkSources_855810_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/7bdba7bc-7091-4bb8-947b-0abdc132ba19_isrForCrosstalkSources_855810_62.stderr
2024-03-09 17:36:02 proc_lsst.multi:146 [INFO]  found job 64340 in provider local
proc_lsst.multi INFO: found job 64340 in provider local
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 372
parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism
parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out
parsl.dataflow.strategy DEBUG: Requesting 92 more blocks
parsl.executors.status_handling INFO: Scaling out by 92 blocks
parsl.executors.status_handling INFO: Allocated block ID 1
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1
2024-03-09 17:36:02 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-1
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-1
2024-03-09 17:36:02 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:02 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-09 17:36:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:02 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:03 proc_lsst.multi:170 [INFO]  [multi] job_id 16966475
proc_lsst.multi INFO: [multi] job_id 16966475
2024-03-09 17:36:03 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:03 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:03 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966475
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966475
parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 16966475
parsl.executors.status_handling INFO: Allocated block ID 2
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2
2024-03-09 17:36:03 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-2
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-2
2024-03-09 17:36:03 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:03 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:03 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:03 proc_lsst.multi:170 [INFO]  [multi] job_id 16966476
proc_lsst.multi INFO: [multi] job_id 16966476
2024-03-09 17:36:03 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-09 17:36:03 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:03 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966476
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966476
parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 16966476
parsl.executors.status_handling INFO: Allocated block ID 3
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3
2024-03-09 17:36:03 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-3
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-3
2024-03-09 17:36:03 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:03 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-09 17:36:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:03 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:04 proc_lsst.multi:170 [INFO]  [multi] job_id 16966477
proc_lsst.multi INFO: [multi] job_id 16966477
2024-03-09 17:36:04 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-09 17:36:04 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:04 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966477
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966477
parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 16966477
parsl.executors.status_handling INFO: Allocated block ID 4
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4
2024-03-09 17:36:04 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-4
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-4
2024-03-09 17:36:04 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:04 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-09 17:36:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:04 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:04 proc_lsst.multi:170 [INFO]  [multi] job_id 16966478
proc_lsst.multi INFO: [multi] job_id 16966478
2024-03-09 17:36:04 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-09 17:36:04 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:04 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966478
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966478
parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 16966478
parsl.executors.status_handling INFO: Allocated block ID 5
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5
2024-03-09 17:36:04 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-5
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-5
2024-03-09 17:36:04 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:04 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-09 17:36:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:04 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:04 proc_lsst.multi:170 [INFO]  [multi] job_id 16966479
proc_lsst.multi INFO: [multi] job_id 16966479
2024-03-09 17:36:04 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:04 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:04 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966479
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966479
parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 16966479
parsl.executors.status_handling INFO: Allocated block ID 6
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6
2024-03-09 17:36:04 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-6
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-6
2024-03-09 17:36:04 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:04 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:04 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:05 proc_lsst.multi:170 [INFO]  [multi] job_id 16966480
proc_lsst.multi INFO: [multi] job_id 16966480
2024-03-09 17:36:05 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-09 17:36:05 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:05 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966480
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966480
parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 16966480
parsl.executors.status_handling INFO: Allocated block ID 7
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7
2024-03-09 17:36:05 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-7
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-7
2024-03-09 17:36:05 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:05 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-09 17:36:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:05 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:06 proc_lsst.multi:170 [INFO]  [multi] job_id 16966481
proc_lsst.multi INFO: [multi] job_id 16966481
2024-03-09 17:36:06 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-09 17:36:06 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:06 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966481
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966481
parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 16966481
parsl.executors.status_handling INFO: Allocated block ID 8
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8
2024-03-09 17:36:06 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-8
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-8
2024-03-09 17:36:06 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:06 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-09 17:36:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:06 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:07 proc_lsst.multi:170 [INFO]  [multi] job_id 16966482
proc_lsst.multi INFO: [multi] job_id 16966482
2024-03-09 17:36:07 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-09 17:36:07 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:07 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966482
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966482
parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 16966482
parsl.executors.status_handling INFO: Allocated block ID 9
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9
2024-03-09 17:36:07 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-9
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-9
2024-03-09 17:36:07 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:07 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-09 17:36:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:07 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:08 proc_lsst.multi:170 [INFO]  [multi] job_id 16966483
proc_lsst.multi INFO: [multi] job_id 16966483
2024-03-09 17:36:08 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-09 17:36:08 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:08 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966483
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966483
parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 16966483
parsl.executors.status_handling INFO: Allocated block ID 10
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10
2024-03-09 17:36:08 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-10
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-10
2024-03-09 17:36:08 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:08 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-09 17:36:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:08 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:09 proc_lsst.multi:170 [INFO]  [multi] job_id 16966484
proc_lsst.multi INFO: [multi] job_id 16966484
2024-03-09 17:36:09 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-09 17:36:09 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:09 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966484
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966484
parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 16966484
parsl.executors.status_handling INFO: Allocated block ID 11
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11
2024-03-09 17:36:09 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-11
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-11
2024-03-09 17:36:09 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:09 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-09 17:36:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:09 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:10 proc_lsst.multi:170 [INFO]  [multi] job_id 16966485
proc_lsst.multi INFO: [multi] job_id 16966485
2024-03-09 17:36:10 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-09 17:36:10 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:10 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966485
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966485
parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 16966485
parsl.executors.status_handling INFO: Allocated block ID 12
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12
2024-03-09 17:36:10 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-12
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-12
2024-03-09 17:36:10 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:10 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-09 17:36:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:10 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:10 proc_lsst.multi:170 [INFO]  [multi] job_id 16966486
proc_lsst.multi INFO: [multi] job_id 16966486
2024-03-09 17:36:10 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-09 17:36:10 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:10 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966486
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966486
parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 16966486
parsl.executors.status_handling INFO: Allocated block ID 13
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13
2024-03-09 17:36:10 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-13
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-13
2024-03-09 17:36:10 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:10 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-09 17:36:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:10 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:10 proc_lsst.multi:170 [INFO]  [multi] job_id 16966487
proc_lsst.multi INFO: [multi] job_id 16966487
2024-03-09 17:36:10 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-09 17:36:10 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:10 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966487
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966487
parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 16966487
parsl.executors.status_handling INFO: Allocated block ID 14
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14
2024-03-09 17:36:10 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-14
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-14
2024-03-09 17:36:10 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:10 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-09 17:36:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:10 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:11 proc_lsst.multi:170 [INFO]  [multi] job_id 16966488
proc_lsst.multi INFO: [multi] job_id 16966488
2024-03-09 17:36:11 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-09 17:36:11 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:11 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966488
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966488
parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 16966488
parsl.executors.status_handling INFO: Allocated block ID 15
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15
2024-03-09 17:36:11 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-15
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-15
2024-03-09 17:36:11 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:11 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-09 17:36:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:11 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:11 proc_lsst.multi:170 [INFO]  [multi] job_id 16966489
proc_lsst.multi INFO: [multi] job_id 16966489
2024-03-09 17:36:11 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-09 17:36:11 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:11 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966489
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966489
parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 16966489
parsl.executors.status_handling INFO: Allocated block ID 16
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16
2024-03-09 17:36:11 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-16
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-16
2024-03-09 17:36:11 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:11 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-09 17:36:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:11 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:11 proc_lsst.multi:170 [INFO]  [multi] job_id 16966490
proc_lsst.multi INFO: [multi] job_id 16966490
2024-03-09 17:36:11 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-09 17:36:11 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:11 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966490
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966490
parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 16966490
parsl.executors.status_handling INFO: Allocated block ID 17
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17
2024-03-09 17:36:11 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-17
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-17
2024-03-09 17:36:11 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:11 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-09 17:36:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:11 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:11 proc_lsst.multi:170 [INFO]  [multi] job_id 16966491
proc_lsst.multi INFO: [multi] job_id 16966491
2024-03-09 17:36:11 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-09 17:36:11 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:11 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966491
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966491
parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 16966491
parsl.executors.status_handling INFO: Allocated block ID 18
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18
2024-03-09 17:36:11 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-18
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-18
2024-03-09 17:36:11 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:11 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-09 17:36:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:11 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:12 proc_lsst.multi:170 [INFO]  [multi] job_id 16966492
proc_lsst.multi INFO: [multi] job_id 16966492
2024-03-09 17:36:12 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-09 17:36:12 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:12 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966492
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966492
parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 16966492
parsl.executors.status_handling INFO: Allocated block ID 19
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19
2024-03-09 17:36:12 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-19
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-19
2024-03-09 17:36:12 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:12 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-09 17:36:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:12 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:12 proc_lsst.multi:170 [INFO]  [multi] job_id 16966493
proc_lsst.multi INFO: [multi] job_id 16966493
2024-03-09 17:36:12 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-09 17:36:12 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:12 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966493
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966493
parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 16966493
parsl.executors.status_handling INFO: Allocated block ID 20
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20
2024-03-09 17:36:12 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-20
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-20
2024-03-09 17:36:12 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:12 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-09 17:36:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:12 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:14 proc_lsst.multi:170 [INFO]  [multi] job_id 16966499
proc_lsst.multi INFO: [multi] job_id 16966499
2024-03-09 17:36:14 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:14 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:14 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966499
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966499
parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 16966499
parsl.executors.status_handling INFO: Allocated block ID 21
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21
2024-03-09 17:36:14 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-21
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-21
2024-03-09 17:36:14 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:14 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:14 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:15 proc_lsst.multi:170 [INFO]  [multi] job_id 16966500
proc_lsst.multi INFO: [multi] job_id 16966500
2024-03-09 17:36:15 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-09 17:36:15 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:15 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966500
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966500
parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 16966500
parsl.executors.status_handling INFO: Allocated block ID 22
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22
2024-03-09 17:36:15 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-22
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-22
2024-03-09 17:36:15 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:15 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:15 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:15 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:15 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-09 17:36:15 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:15 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:15 proc_lsst.multi:170 [INFO]  [multi] job_id 16966501
proc_lsst.multi INFO: [multi] job_id 16966501
2024-03-09 17:36:15 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-09 17:36:15 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:15 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966501
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966501
parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 16966501
parsl.executors.status_handling INFO: Allocated block ID 23
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23
2024-03-09 17:36:15 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-23
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-23
2024-03-09 17:36:15 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:15 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:15 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:15 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:15 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-09 17:36:15 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:15 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:16 proc_lsst.multi:170 [INFO]  [multi] job_id 16966502
proc_lsst.multi INFO: [multi] job_id 16966502
2024-03-09 17:36:16 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-09 17:36:16 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:16 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966502
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966502
parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 16966502
parsl.executors.status_handling INFO: Allocated block ID 24
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24
2024-03-09 17:36:16 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-24
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-24
2024-03-09 17:36:16 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:16 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:16 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:16 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:16 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-09 17:36:16 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:16 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:16 proc_lsst.multi:170 [INFO]  [multi] job_id 16966503
proc_lsst.multi INFO: [multi] job_id 16966503
2024-03-09 17:36:16 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-09 17:36:16 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:16 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966503
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966503
parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 16966503
parsl.executors.status_handling INFO: Allocated block ID 25
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25
2024-03-09 17:36:16 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-25
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-25
2024-03-09 17:36:16 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:16 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:16 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:16 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:16 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-09 17:36:16 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:16 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:17 proc_lsst.multi:170 [INFO]  [multi] job_id 16966504
proc_lsst.multi INFO: [multi] job_id 16966504
2024-03-09 17:36:17 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-09 17:36:17 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:17 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966504
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966504
parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 16966504
parsl.executors.status_handling INFO: Allocated block ID 26
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26
2024-03-09 17:36:17 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-26
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-26
2024-03-09 17:36:17 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:17 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:17 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:17 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:17 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-09 17:36:17 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:17 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 40ae28ba7217b909e7ed07471576ff06 with result from task 0
parsl.dataflow.dflow DEBUG: Task 0 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/09eebe30-3ada-4057-ba9e-2978d0fcfd43_isrForCrosstalkSources_855717_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/09eebe30-3ada-4057-ba9e-2978d0fcfd43_isrForCrosstalkSources_855717_27.stderr
2024-03-09 17:36:18 proc_lsst.multi:170 [INFO]  [multi] job_id 16966505
proc_lsst.multi INFO: [multi] job_id 16966505
2024-03-09 17:36:18 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-09 17:36:18 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:18 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966505
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966505
parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 16966505
parsl.executors.status_handling INFO: Allocated block ID 27
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27
2024-03-09 17:36:18 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-27
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-27
2024-03-09 17:36:18 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:18 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:18 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:18 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:18 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-09 17:36:18 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:18 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 649ad4f0da844d917cf4e521ade6a90e with result from task 1
parsl.dataflow.dflow DEBUG: Task 1 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/18f4ef95-1342-4234-8b2e-434e20665e5c_isrForCrosstalkSources_855716_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/18f4ef95-1342-4234-8b2e-434e20665e5c_isrForCrosstalkSources_855716_11.stderr
2024-03-09 17:36:18 proc_lsst.multi:170 [INFO]  [multi] job_id 16966506
proc_lsst.multi INFO: [multi] job_id 16966506
2024-03-09 17:36:18 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-09 17:36:18 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:18 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966506
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966506
parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 16966506
parsl.executors.status_handling INFO: Allocated block ID 28
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28
2024-03-09 17:36:18 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-28
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-28
2024-03-09 17:36:18 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:18 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:18 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:18 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:18 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-09 17:36:18 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:18 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:18 proc_lsst.multi:170 [INFO]  [multi] job_id 16966507
proc_lsst.multi INFO: [multi] job_id 16966507
2024-03-09 17:36:18 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-09 17:36:18 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:18 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966507
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966507
parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 16966507
parsl.executors.status_handling INFO: Allocated block ID 29
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29
2024-03-09 17:36:18 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-29
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-29
2024-03-09 17:36:18 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:18 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:18 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:18 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:18 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-09 17:36:18 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:18 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:19 proc_lsst.multi:170 [INFO]  [multi] job_id 16966508
proc_lsst.multi INFO: [multi] job_id 16966508
2024-03-09 17:36:19 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-09 17:36:19 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:19 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966508
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966508
parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 16966508
parsl.executors.status_handling INFO: Allocated block ID 30
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30
2024-03-09 17:36:19 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-30
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-30
2024-03-09 17:36:19 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:19 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:19 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:19 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:19 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-09 17:36:19 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:19 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:19 proc_lsst.multi:170 [INFO]  [multi] job_id 16966509
proc_lsst.multi INFO: [multi] job_id 16966509
2024-03-09 17:36:19 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:19 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:19 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16966509
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16966509
parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 16966509
parsl.executors.status_handling INFO: Allocated block ID 31
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31
2024-03-09 17:36:19 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-31
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-31
2024-03-09 17:36:19 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:19 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:19 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:19 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:19 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:19 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:19 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:19 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-09 17:36:19 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:19 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:20 proc_lsst.multi:170 [INFO]  [multi] job_id 16966510
proc_lsst.multi INFO: [multi] job_id 16966510
2024-03-09 17:36:20 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:20 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:20 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 16966510
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16966510
parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 16966510
parsl.executors.status_handling INFO: Allocated block ID 32
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32
2024-03-09 17:36:20 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-32
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-32
2024-03-09 17:36:20 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:20 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:20 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:20 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:20 proc_lsst.multi:170 [INFO]  [multi] job_id 16966511
proc_lsst.multi INFO: [multi] job_id 16966511
2024-03-09 17:36:20 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-09 17:36:20 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:20 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 16966511
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16966511
parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 16966511
parsl.executors.status_handling INFO: Allocated block ID 33
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33
2024-03-09 17:36:20 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-33
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-33
2024-03-09 17:36:20 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:20 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:20 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-09 17:36:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:20 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:21 proc_lsst.multi:170 [INFO]  [multi] job_id 16966512
proc_lsst.multi INFO: [multi] job_id 16966512
2024-03-09 17:36:21 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-09 17:36:21 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:21 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 16966512
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16966512
parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 16966512
parsl.executors.status_handling INFO: Allocated block ID 34
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34
2024-03-09 17:36:21 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-34
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-34
2024-03-09 17:36:21 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:21 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:21 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:21 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:21 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:21 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:21 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:21 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-09 17:36:21 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:21 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:21 proc_lsst.multi:170 [INFO]  [multi] job_id 16966513
proc_lsst.multi INFO: [multi] job_id 16966513
2024-03-09 17:36:21 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-09 17:36:21 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:21 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 16966513
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16966513
parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 16966513
parsl.executors.status_handling INFO: Allocated block ID 35
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35
2024-03-09 17:36:21 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-35
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-35
2024-03-09 17:36:21 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:21 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:21 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:21 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:21 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:21 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:21 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:21 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-09 17:36:21 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:21 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:22 proc_lsst.multi:170 [INFO]  [multi] job_id 16966514
proc_lsst.multi INFO: [multi] job_id 16966514
2024-03-09 17:36:22 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:22 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:22 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 16966514
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16966514
parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 16966514
parsl.executors.status_handling INFO: Allocated block ID 36
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36
2024-03-09 17:36:22 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-36
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-36
2024-03-09 17:36:22 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:22 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:22 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:22 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:22 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:22 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:22 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:22 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:22 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:22 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:22 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-09 17:36:22 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:22 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:23 proc_lsst.multi:170 [INFO]  [multi] job_id 16966515
proc_lsst.multi INFO: [multi] job_id 16966515
2024-03-09 17:36:23 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:23 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:23 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16966515
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16966515
parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 16966515
parsl.executors.status_handling INFO: Allocated block ID 37
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37
2024-03-09 17:36:23 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-37
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-37
2024-03-09 17:36:23 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:23 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:23 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:23 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:23 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:23 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:23 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:23 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:23 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:23 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:23 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:23 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:23 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:23 proc_lsst.multi:170 [INFO]  [multi] job_id 16966516
proc_lsst.multi INFO: [multi] job_id 16966516
2024-03-09 17:36:23 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-09 17:36:23 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:23 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16966516
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16966516
parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 16966516
parsl.executors.status_handling INFO: Allocated block ID 38
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38
2024-03-09 17:36:23 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-38
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-38
2024-03-09 17:36:23 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:23 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:23 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:23 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:23 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:23 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:23 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:23 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:23 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:23 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:23 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-09 17:36:23 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:23 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:24 proc_lsst.multi:170 [INFO]  [multi] job_id 16966517
proc_lsst.multi INFO: [multi] job_id 16966517
2024-03-09 17:36:24 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-09 17:36:24 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:24 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16966517
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16966517
parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 16966517
parsl.executors.status_handling INFO: Allocated block ID 39
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39
2024-03-09 17:36:24 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-39
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-39
2024-03-09 17:36:24 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:24 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:24 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:24 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:24 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:24 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:24 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:24 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:24 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:24 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:24 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-09 17:36:24 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:24 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:24 proc_lsst.multi:170 [INFO]  [multi] job_id 16966518
proc_lsst.multi INFO: [multi] job_id 16966518
2024-03-09 17:36:24 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-09 17:36:24 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:24 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16966518
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16966518
parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 16966518
parsl.executors.status_handling INFO: Allocated block ID 40
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40
2024-03-09 17:36:24 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-40
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-40
2024-03-09 17:36:24 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:24 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:24 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:24 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:24 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:24 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:24 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:24 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:24 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:24 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:24 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-09 17:36:24 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:24 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:24 proc_lsst.multi:170 [INFO]  [multi] job_id 16966519
proc_lsst.multi INFO: [multi] job_id 16966519
2024-03-09 17:36:24 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:24 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:24 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16966519
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16966519
parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 16966519
parsl.executors.status_handling INFO: Allocated block ID 41
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41
2024-03-09 17:36:24 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-41
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-41
2024-03-09 17:36:24 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:24 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:24 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:24 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:24 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:24 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:24 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:24 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:24 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:24 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:24 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:24 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:24 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:25 proc_lsst.multi:170 [INFO]  [multi] job_id 16966520
proc_lsst.multi INFO: [multi] job_id 16966520
2024-03-09 17:36:25 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-09 17:36:25 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:25 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16966520
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16966520
parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 16966520
parsl.executors.status_handling INFO: Allocated block ID 42
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42
2024-03-09 17:36:25 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-42
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-42
2024-03-09 17:36:25 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:25 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:25 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:25 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:25 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:25 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:25 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:25 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:25 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:25 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:25 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-09 17:36:25 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:25 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:25 proc_lsst.multi:170 [INFO]  [multi] job_id 16966521
proc_lsst.multi INFO: [multi] job_id 16966521
2024-03-09 17:36:25 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-09 17:36:25 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:25 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16966521
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16966521
parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 16966521
parsl.executors.status_handling INFO: Allocated block ID 43
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43
2024-03-09 17:36:25 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-43
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-43
2024-03-09 17:36:25 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:25 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:25 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:25 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:25 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:25 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:25 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:25 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:25 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:25 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:25 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-09 17:36:25 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:25 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6f8cbd26c5e2e9dc3ad04e9f437990d8 with result from task 2
parsl.dataflow.dflow DEBUG: Task 2 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/55d4de18-27a0-4dc6-9e6a-0425e4e561cb_isrForCrosstalkSources_855810_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/55d4de18-27a0-4dc6-9e6a-0425e4e561cb_isrForCrosstalkSources_855810_61.stderr
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:26 proc_lsst.multi:170 [INFO]  [multi] job_id 16966522
proc_lsst.multi INFO: [multi] job_id 16966522
2024-03-09 17:36:26 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-09 17:36:26 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:26 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16966522
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16966522
parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 16966522
parsl.executors.status_handling INFO: Allocated block ID 44
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44
2024-03-09 17:36:26 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-44
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-44
2024-03-09 17:36:26 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:26 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:26 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:26 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:26 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:26 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:26 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:26 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:26 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:26 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:26 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-09 17:36:26 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:26 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:26 proc_lsst.multi:170 [INFO]  [multi] job_id 16966523
proc_lsst.multi INFO: [multi] job_id 16966523
2024-03-09 17:36:26 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-09 17:36:26 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:26 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16966523
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16966523
parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 16966523
parsl.executors.status_handling INFO: Allocated block ID 45
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45
2024-03-09 17:36:26 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-45
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-45
2024-03-09 17:36:26 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:26 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:26 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:26 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:26 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:26 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:26 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:26 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:26 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:26 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:26 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-09 17:36:26 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:26 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:27 proc_lsst.multi:170 [INFO]  [multi] job_id 16966524
proc_lsst.multi INFO: [multi] job_id 16966524
2024-03-09 17:36:27 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-09 17:36:27 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:27 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16966524
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16966524
parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 16966524
parsl.executors.status_handling INFO: Allocated block ID 46
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46
2024-03-09 17:36:27 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-46
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-46
2024-03-09 17:36:27 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:27 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:27 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:27 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:27 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:27 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:27 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:27 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:27 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:27 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:27 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-09 17:36:27 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:27 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:27 proc_lsst.multi:170 [INFO]  [multi] job_id 16966525
proc_lsst.multi INFO: [multi] job_id 16966525
2024-03-09 17:36:27 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-09 17:36:27 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:27 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16966525
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16966525
parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 16966525
parsl.executors.status_handling INFO: Allocated block ID 47
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47
2024-03-09 17:36:27 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-47
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-47
2024-03-09 17:36:27 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:27 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:27 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:27 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:27 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:27 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:27 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:27 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:27 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:27 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:27 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-09 17:36:27 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:27 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:28 proc_lsst.multi:170 [INFO]  [multi] job_id 16966526
proc_lsst.multi INFO: [multi] job_id 16966526
2024-03-09 17:36:28 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-09 17:36:28 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:28 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16966526
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16966526
parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 16966526
parsl.executors.status_handling INFO: Allocated block ID 48
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48
2024-03-09 17:36:28 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-48
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-48
2024-03-09 17:36:28 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:28 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:28 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:28 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-09 17:36:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:28 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:28 proc_lsst.multi:170 [INFO]  [multi] job_id 16966527
proc_lsst.multi INFO: [multi] job_id 16966527
2024-03-09 17:36:28 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-09 17:36:28 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:28 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16966527
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16966527
parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 16966527
parsl.executors.status_handling INFO: Allocated block ID 49
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49
2024-03-09 17:36:28 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-49
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-49
2024-03-09 17:36:28 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:28 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:28 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:28 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-09 17:36:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:28 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:29 proc_lsst.multi:170 [INFO]  [multi] job_id 16966528
proc_lsst.multi INFO: [multi] job_id 16966528
2024-03-09 17:36:29 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-09 17:36:29 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:29 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16966528
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16966528
parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 16966528
parsl.executors.status_handling INFO: Allocated block ID 50
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50
2024-03-09 17:36:29 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-50
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-50
2024-03-09 17:36:29 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:29 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:29 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:29 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-09 17:36:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:29 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:30 proc_lsst.multi:170 [INFO]  [multi] job_id 16966529
proc_lsst.multi INFO: [multi] job_id 16966529
2024-03-09 17:36:30 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-09 17:36:30 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:30 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16966529
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16966529
parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 16966529
parsl.executors.status_handling INFO: Allocated block ID 51
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51
2024-03-09 17:36:30 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-51
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-51
2024-03-09 17:36:30 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:30 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:30 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:30 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-09 17:36:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:30 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 29444492f0761a46a8ffc5e0d8a81681 with result from task 4
parsl.dataflow.dflow DEBUG: Task 4 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/8f0db560-94cd-458d-b4fa-d35fc1765b8f_isrForCrosstalkSources_855716_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/8f0db560-94cd-458d-b4fa-d35fc1765b8f_isrForCrosstalkSources_855716_14.stderr
2024-03-09 17:36:30 proc_lsst.multi:170 [INFO]  [multi] job_id 16966530
proc_lsst.multi INFO: [multi] job_id 16966530
2024-03-09 17:36:30 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-09 17:36:30 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:30 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16966530
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16966530
parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 16966530
parsl.executors.status_handling INFO: Allocated block ID 52
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52
2024-03-09 17:36:30 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-52
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-52
2024-03-09 17:36:30 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:30 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:30 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:30 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-09 17:36:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:30 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6ad2440a4d3bf37575e1dc63a6145474 with result from task 5
parsl.dataflow.dflow DEBUG: Task 5 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/1de5d971-c596-4702-aed8-0725b51cc004_isrForCrosstalkSources_855810_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/1de5d971-c596-4702-aed8-0725b51cc004_isrForCrosstalkSources_855810_45.stderr
parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5c2f77601f07b2bde362631adb35c751 with result from task 6
parsl.dataflow.dflow DEBUG: Task 6 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/46c8b1d8-d4cd-48ba-be6b-017d6567523b_isrForCrosstalkSources_855810_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/46c8b1d8-d4cd-48ba-be6b-017d6567523b_isrForCrosstalkSources_855810_31.stderr
parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9963a0312bdda7f9920651b42c2c5105 with result from task 3
parsl.dataflow.dflow DEBUG: Task 3 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/95945658-1a14-4d93-9333-f84361dc4358_isrForCrosstalkSources_855717_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/95945658-1a14-4d93-9333-f84361dc4358_isrForCrosstalkSources_855717_1.stderr
2024-03-09 17:36:32 proc_lsst.multi:170 [INFO]  [multi] job_id 16966531
proc_lsst.multi INFO: [multi] job_id 16966531
2024-03-09 17:36:32 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-09 17:36:32 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:32 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16966531
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16966531
parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 16966531
parsl.executors.status_handling INFO: Allocated block ID 53
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53
2024-03-09 17:36:32 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-53
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-53
2024-03-09 17:36:32 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:32 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:32 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:32 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:32 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:32 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:32 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:32 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:32 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:32 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:32 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-09 17:36:32 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:32 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:33 proc_lsst.multi:170 [INFO]  [multi] job_id 16966532
proc_lsst.multi INFO: [multi] job_id 16966532
2024-03-09 17:36:33 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-09 17:36:33 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:33 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16966532
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16966532
parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 16966532
parsl.executors.status_handling INFO: Allocated block ID 54
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54
2024-03-09 17:36:33 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-54
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-54
2024-03-09 17:36:33 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:33 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:33 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:33 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-09 17:36:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:33 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:33 proc_lsst.multi:170 [INFO]  [multi] job_id 16966533
proc_lsst.multi INFO: [multi] job_id 16966533
2024-03-09 17:36:33 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-09 17:36:33 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:33 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16966533
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16966533
parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 16966533
parsl.executors.status_handling INFO: Allocated block ID 55
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55
2024-03-09 17:36:33 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-55
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-55
2024-03-09 17:36:33 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:33 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:33 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:33 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-09 17:36:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:33 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:34 proc_lsst.multi:170 [INFO]  [multi] job_id 16966534
proc_lsst.multi INFO: [multi] job_id 16966534
2024-03-09 17:36:34 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:34 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:34 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16966534
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16966534
parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 16966534
parsl.executors.status_handling INFO: Allocated block ID 56
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56
2024-03-09 17:36:34 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-56
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-56
2024-03-09 17:36:34 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:34 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:34 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:34 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:34 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-09 17:36:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:34 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:34 proc_lsst.multi:170 [INFO]  [multi] job_id 16966535
proc_lsst.multi INFO: [multi] job_id 16966535
2024-03-09 17:36:34 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:34 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:34 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966535
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966535
parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 16966535
parsl.executors.status_handling INFO: Allocated block ID 57
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57
2024-03-09 17:36:34 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-57
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-57
2024-03-09 17:36:34 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:34 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:34 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:34 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:34 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:34 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:35 proc_lsst.multi:170 [INFO]  [multi] job_id 16966536
proc_lsst.multi INFO: [multi] job_id 16966536
2024-03-09 17:36:35 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-09 17:36:35 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:35 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966536
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966536
parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 16966536
parsl.executors.status_handling INFO: Allocated block ID 58
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58
2024-03-09 17:36:35 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-58
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-58
2024-03-09 17:36:35 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:35 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:35 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:35 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:35 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-09 17:36:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:35 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:35 proc_lsst.multi:170 [INFO]  [multi] job_id 16966537
proc_lsst.multi INFO: [multi] job_id 16966537
2024-03-09 17:36:35 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-09 17:36:35 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:35 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966537
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966537
parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 16966537
parsl.executors.status_handling INFO: Allocated block ID 59
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59
2024-03-09 17:36:35 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-59
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-59
2024-03-09 17:36:35 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:35 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:35 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:35 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:35 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-09 17:36:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:35 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:36 proc_lsst.multi:170 [INFO]  [multi] job_id 16966538
proc_lsst.multi INFO: [multi] job_id 16966538
2024-03-09 17:36:36 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-09 17:36:36 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:36 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966538
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966538
parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 16966538
parsl.executors.status_handling INFO: Allocated block ID 60
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60
2024-03-09 17:36:36 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-60
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-60
2024-03-09 17:36:36 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:36 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:36 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:36 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:36 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-09 17:36:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:36 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:36 proc_lsst.multi:170 [INFO]  [multi] job_id 16966539
proc_lsst.multi INFO: [multi] job_id 16966539
2024-03-09 17:36:36 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:36 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:36 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966539
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966539
parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 16966539
parsl.executors.status_handling INFO: Allocated block ID 61
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61
2024-03-09 17:36:36 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-61
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-61
2024-03-09 17:36:36 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:36 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:36 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:36 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:36 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:36 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:37 proc_lsst.multi:170 [INFO]  [multi] job_id 16966540
proc_lsst.multi INFO: [multi] job_id 16966540
2024-03-09 17:36:37 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-09 17:36:37 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:37 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966540
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966540
parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 16966540
parsl.executors.status_handling INFO: Allocated block ID 62
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62
2024-03-09 17:36:37 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-62
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-62
2024-03-09 17:36:37 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:37 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:37 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:37 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:37 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-09 17:36:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:37 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6903e1968552ef172382d67b0c92dfa0 with result from task 34
parsl.dataflow.dflow DEBUG: Task 34 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/c8f50f66-92ba-441d-849f-717b50ee6f9c_isrForCrosstalkSources_855810_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/c8f50f66-92ba-441d-849f-717b50ee6f9c_isrForCrosstalkSources_855810_24.stderr
2024-03-09 17:36:37 proc_lsst.multi:170 [INFO]  [multi] job_id 16966541
proc_lsst.multi INFO: [multi] job_id 16966541
2024-03-09 17:36:37 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-09 17:36:37 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:37 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966541
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966541
parsl.executors.status_handling DEBUG: Launched block 62 on executor multi with job ID 16966541
parsl.executors.status_handling INFO: Allocated block ID 63
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63
2024-03-09 17:36:37 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-63
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-63
2024-03-09 17:36:37 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:37 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:37 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:37 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:37 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-09 17:36:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:37 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:38 proc_lsst.multi:170 [INFO]  [multi] job_id 16966542
proc_lsst.multi INFO: [multi] job_id 16966542
2024-03-09 17:36:38 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-09 17:36:38 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:38 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966542
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966542
parsl.executors.status_handling DEBUG: Launched block 63 on executor multi with job ID 16966542
parsl.executors.status_handling INFO: Allocated block ID 64
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64
2024-03-09 17:36:38 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-64
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-64
2024-03-09 17:36:38 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:38 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:38 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:38 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:38 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-09 17:36:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:38 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:38 proc_lsst.multi:170 [INFO]  [multi] job_id 16966543
proc_lsst.multi INFO: [multi] job_id 16966543
2024-03-09 17:36:38 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-09 17:36:38 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:38 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966543
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966543
parsl.executors.status_handling DEBUG: Launched block 64 on executor multi with job ID 16966543
parsl.executors.status_handling INFO: Allocated block ID 65
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65
2024-03-09 17:36:38 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-65
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-65
2024-03-09 17:36:38 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:38 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:38 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:38 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:38 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-09 17:36:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:38 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:39 proc_lsst.multi:170 [INFO]  [multi] job_id 16966544
proc_lsst.multi INFO: [multi] job_id 16966544
2024-03-09 17:36:39 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-09 17:36:39 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:39 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966544
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966544
parsl.executors.status_handling DEBUG: Launched block 65 on executor multi with job ID 16966544
parsl.executors.status_handling INFO: Allocated block ID 66
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66
2024-03-09 17:36:39 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-66
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-66
2024-03-09 17:36:39 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:39 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:39 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:39 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:39 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-09 17:36:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:39 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:41 proc_lsst.multi:170 [INFO]  [multi] job_id 16966545
proc_lsst.multi INFO: [multi] job_id 16966545
2024-03-09 17:36:41 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-09 17:36:41 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:41 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966545
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966545
parsl.executors.status_handling DEBUG: Launched block 66 on executor multi with job ID 16966545
parsl.executors.status_handling INFO: Allocated block ID 67
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67
2024-03-09 17:36:41 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-67
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-67
2024-03-09 17:36:41 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:41 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:41 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:41 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:41 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-09 17:36:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:41 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:42 proc_lsst.multi:170 [INFO]  [multi] job_id 16966546
proc_lsst.multi INFO: [multi] job_id 16966546
2024-03-09 17:36:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-09 17:36:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:42 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966546
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966546
parsl.executors.status_handling DEBUG: Launched block 67 on executor multi with job ID 16966546
parsl.executors.status_handling INFO: Allocated block ID 68
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68
2024-03-09 17:36:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-68
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-68
2024-03-09 17:36:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:42 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:42 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:42 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-09 17:36:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:42 proc_lsst.multi:170 [INFO]  [multi] job_id 16966547
proc_lsst.multi INFO: [multi] job_id 16966547
2024-03-09 17:36:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-09 17:36:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:42 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966547
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966547
parsl.executors.status_handling DEBUG: Launched block 68 on executor multi with job ID 16966547
parsl.executors.status_handling INFO: Allocated block ID 69
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69
2024-03-09 17:36:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-69
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-69
2024-03-09 17:36:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:42 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:42 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:42 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-09 17:36:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:43 proc_lsst.multi:170 [INFO]  [multi] job_id 16966548
proc_lsst.multi INFO: [multi] job_id 16966548
2024-03-09 17:36:43 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-09 17:36:43 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:43 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966548
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966548
parsl.executors.status_handling DEBUG: Launched block 69 on executor multi with job ID 16966548
parsl.executors.status_handling INFO: Allocated block ID 70
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70
2024-03-09 17:36:43 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-70
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-70
2024-03-09 17:36:43 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:43 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:43 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:43 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:43 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-09 17:36:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:43 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a57146dd734ac999f4b8ed02bb398a57 with result from task 36
parsl.dataflow.dflow DEBUG: Task 36 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/794c6658-c57a-450d-a8ce-07caad22a049_isrForCrosstalkSources_855717_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/794c6658-c57a-450d-a8ce-07caad22a049_isrForCrosstalkSources_855717_44.stderr
2024-03-09 17:36:43 proc_lsst.multi:170 [INFO]  [multi] job_id 16966549
proc_lsst.multi INFO: [multi] job_id 16966549
2024-03-09 17:36:43 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-09 17:36:43 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:43 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966549
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966549
parsl.executors.status_handling DEBUG: Launched block 70 on executor multi with job ID 16966549
parsl.executors.status_handling INFO: Allocated block ID 71
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71
2024-03-09 17:36:43 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-71
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-71
2024-03-09 17:36:43 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:43 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:43 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:43 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:43 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-09 17:36:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:43 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 15ebb43bfd00563b6d95baa69bf07f18 with result from task 37
parsl.dataflow.dflow DEBUG: Task 37 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/d0fdc23a-b571-4fd0-8437-7035ee877a26_isrForCrosstalkSources_855717_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/d0fdc23a-b571-4fd0-8437-7035ee877a26_isrForCrosstalkSources_855717_54.stderr
2024-03-09 17:36:44 proc_lsst.multi:170 [INFO]  [multi] job_id 16966550
proc_lsst.multi INFO: [multi] job_id 16966550
2024-03-09 17:36:44 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-09 17:36:44 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:44 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966550
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966550
parsl.executors.status_handling DEBUG: Launched block 71 on executor multi with job ID 16966550
parsl.executors.status_handling INFO: Allocated block ID 72
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72
2024-03-09 17:36:44 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-72
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-72
2024-03-09 17:36:44 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:44 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:44 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:44 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:44 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-09 17:36:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:44 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:44 proc_lsst.multi:170 [INFO]  [multi] job_id 16966551
proc_lsst.multi INFO: [multi] job_id 16966551
2024-03-09 17:36:44 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-09 17:36:44 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:44 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966551
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966551
parsl.executors.status_handling DEBUG: Launched block 72 on executor multi with job ID 16966551
parsl.executors.status_handling INFO: Allocated block ID 73
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73
2024-03-09 17:36:44 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-73
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-73
2024-03-09 17:36:44 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:44 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:44 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:44 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:44 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-09 17:36:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:44 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:46 proc_lsst.multi:170 [INFO]  [multi] job_id 16966553
proc_lsst.multi INFO: [multi] job_id 16966553
2024-03-09 17:36:46 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-09 17:36:46 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:46 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966553
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966553
parsl.executors.status_handling DEBUG: Launched block 73 on executor multi with job ID 16966553
parsl.executors.status_handling INFO: Allocated block ID 74
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74
2024-03-09 17:36:46 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-74
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-74
2024-03-09 17:36:46 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:46 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:46 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:46 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:46 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-09 17:36:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:46 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:46 proc_lsst.multi:170 [INFO]  [multi] job_id 16966554
proc_lsst.multi INFO: [multi] job_id 16966554
2024-03-09 17:36:46 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-09 17:36:46 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:46 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966554
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966554
parsl.executors.status_handling DEBUG: Launched block 74 on executor multi with job ID 16966554
parsl.executors.status_handling INFO: Allocated block ID 75
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75
2024-03-09 17:36:46 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-75
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-75
2024-03-09 17:36:46 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:46 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:46 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:46 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:46 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-09 17:36:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:46 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:47 proc_lsst.multi:170 [INFO]  [multi] job_id 16966555
proc_lsst.multi INFO: [multi] job_id 16966555
2024-03-09 17:36:47 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:47 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:47 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966555
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966555
parsl.executors.status_handling DEBUG: Launched block 75 on executor multi with job ID 16966555
parsl.executors.status_handling INFO: Allocated block ID 76
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76
2024-03-09 17:36:47 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-76
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-76
2024-03-09 17:36:47 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:47 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:47 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:47 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:47 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:47 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:47 proc_lsst.multi:170 [INFO]  [multi] job_id 16966556
proc_lsst.multi INFO: [multi] job_id 16966556
2024-03-09 17:36:47 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-09 17:36:47 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:47 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966556
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966556
parsl.executors.status_handling DEBUG: Launched block 76 on executor multi with job ID 16966556
parsl.executors.status_handling INFO: Allocated block ID 77
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77
2024-03-09 17:36:47 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-77
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-77
2024-03-09 17:36:47 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:47 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:47 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:47 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:47 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-09 17:36:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:47 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:47 proc_lsst.multi:170 [INFO]  [multi] job_id 16966557
proc_lsst.multi INFO: [multi] job_id 16966557
2024-03-09 17:36:47 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-09 17:36:47 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:47 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966557
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966557
parsl.executors.status_handling DEBUG: Launched block 77 on executor multi with job ID 16966557
parsl.executors.status_handling INFO: Allocated block ID 78
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-78
2024-03-09 17:36:47 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-78
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-78
2024-03-09 17:36:47 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:47 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:47 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:47 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:47 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-09 17:36:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:47 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:48 proc_lsst.multi:170 [INFO]  [multi] job_id 16966558
proc_lsst.multi INFO: [multi] job_id 16966558
2024-03-09 17:36:48 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-09 17:36:48 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:48 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966558
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966558
parsl.executors.status_handling DEBUG: Launched block 78 on executor multi with job ID 16966558
parsl.executors.status_handling INFO: Allocated block ID 79
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-79
2024-03-09 17:36:48 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-79
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-79
2024-03-09 17:36:48 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:48 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:48 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:48 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:48 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-09 17:36:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:48 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:48 proc_lsst.multi:170 [INFO]  [multi] job_id 16966559
proc_lsst.multi INFO: [multi] job_id 16966559
2024-03-09 17:36:48 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-09 17:36:48 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:48 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966559
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966559
parsl.executors.status_handling DEBUG: Launched block 79 on executor multi with job ID 16966559
parsl.executors.status_handling INFO: Allocated block ID 80
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-80
2024-03-09 17:36:48 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-80
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-80
2024-03-09 17:36:48 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:48 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:48 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:48 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:48 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-09 17:36:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:48 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:48 proc_lsst.multi:170 [INFO]  [multi] job_id 16966560
proc_lsst.multi INFO: [multi] job_id 16966560
2024-03-09 17:36:48 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-09 17:36:48 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:48 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966560
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966560
parsl.executors.status_handling DEBUG: Launched block 80 on executor multi with job ID 16966560
parsl.executors.status_handling INFO: Allocated block ID 81
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-81
2024-03-09 17:36:48 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-81
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-81
2024-03-09 17:36:48 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:48 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:48 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:48 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:48 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-09 17:36:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:48 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:49 proc_lsst.multi:170 [INFO]  [multi] job_id 16966561
proc_lsst.multi INFO: [multi] job_id 16966561
2024-03-09 17:36:49 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-09 17:36:49 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:49 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966561
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966561
parsl.executors.status_handling DEBUG: Launched block 81 on executor multi with job ID 16966561
parsl.executors.status_handling INFO: Allocated block ID 82
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-82
2024-03-09 17:36:49 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-82
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-82
2024-03-09 17:36:49 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:49 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:49 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:49 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:49 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-09 17:36:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:49 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bddd505cabb1a2c3370c2e12a37cbeaf with result from task 49
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow DEBUG: Task 49 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/aa2bd463-71db-4003-b494-ca120733e8c5_isrForCrosstalkSources_855716_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/aa2bd463-71db-4003-b494-ca120733e8c5_isrForCrosstalkSources_855716_16.stderr
parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d36febe6616ff2ad90197bf2e08e79a7 with result from task 35
parsl.dataflow.dflow DEBUG: Task 35 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/0dca4b0a-abac-4a96-ad25-641fe04e4418_isrForCrosstalkSources_855809_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/0dca4b0a-abac-4a96-ad25-641fe04e4418_isrForCrosstalkSources_855809_52.stderr
2024-03-09 17:36:49 proc_lsst.multi:170 [INFO]  [multi] job_id 16966562
proc_lsst.multi INFO: [multi] job_id 16966562
2024-03-09 17:36:49 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-09 17:36:49 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:49 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966562
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966562
parsl.executors.status_handling DEBUG: Launched block 82 on executor multi with job ID 16966562
parsl.executors.status_handling INFO: Allocated block ID 83
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-83
2024-03-09 17:36:49 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-83
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-83
2024-03-09 17:36:49 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:49 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:49 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:49 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:49 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-09 17:36:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:49 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e72f617fac7d751c98318235f62f34ad with result from task 38
parsl.dataflow.dflow DEBUG: Task 38 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/ee41f1a9-30f4-400b-910c-26af5e7cf048_isrForCrosstalkSources_855718_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/ee41f1a9-30f4-400b-910c-26af5e7cf048_isrForCrosstalkSources_855718_61.stderr
2024-03-09 17:36:50 proc_lsst.multi:170 [INFO]  [multi] job_id 16966563
proc_lsst.multi INFO: [multi] job_id 16966563
2024-03-09 17:36:50 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-09 17:36:50 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:50 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966563
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966563
parsl.executors.status_handling DEBUG: Launched block 83 on executor multi with job ID 16966563
parsl.executors.status_handling INFO: Allocated block ID 84
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-84
2024-03-09 17:36:50 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-84
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-84
2024-03-09 17:36:50 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:50 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:50 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:50 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:50 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-09 17:36:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:50 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:50 proc_lsst.multi:170 [INFO]  [multi] job_id 16966564
proc_lsst.multi INFO: [multi] job_id 16966564
2024-03-09 17:36:50 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-09 17:36:50 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:50 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966564
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966564
parsl.executors.status_handling DEBUG: Launched block 84 on executor multi with job ID 16966564
parsl.executors.status_handling INFO: Allocated block ID 85
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-85
2024-03-09 17:36:50 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-85
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-85
2024-03-09 17:36:50 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:50 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:50 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:50 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:50 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-09 17:36:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:50 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:51 proc_lsst.multi:170 [INFO]  [multi] job_id 16966565
proc_lsst.multi INFO: [multi] job_id 16966565
2024-03-09 17:36:51 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:51 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:51 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966565
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966565
parsl.executors.status_handling DEBUG: Launched block 85 on executor multi with job ID 16966565
parsl.executors.status_handling INFO: Allocated block ID 86
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-86
2024-03-09 17:36:51 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-86
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-86
2024-03-09 17:36:51 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:51 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:51 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:51 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:51 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:51 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:51 proc_lsst.multi:170 [INFO]  [multi] job_id 16966566
proc_lsst.multi INFO: [multi] job_id 16966566
2024-03-09 17:36:51 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 31
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31
2024-03-09 17:36:51 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:51 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966566
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966566
parsl.executors.status_handling DEBUG: Launched block 86 on executor multi with job ID 16966566
parsl.executors.status_handling INFO: Allocated block ID 87
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-87
2024-03-09 17:36:51 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-87
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-87
2024-03-09 17:36:51 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:51 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:51 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:51 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:51 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 31
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31
2024-03-09 17:36:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:51 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:52 proc_lsst.multi:170 [INFO]  [multi] job_id 16966567
proc_lsst.multi INFO: [multi] job_id 16966567
2024-03-09 17:36:52 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 32
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32
2024-03-09 17:36:52 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:52 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966567
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966567
parsl.executors.status_handling DEBUG: Launched block 87 on executor multi with job ID 16966567
parsl.executors.status_handling INFO: Allocated block ID 88
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-88
2024-03-09 17:36:52 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-88
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-88
2024-03-09 17:36:52 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:52 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:52 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:52 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:52 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 32
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32
2024-03-09 17:36:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:52 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:52 proc_lsst.multi:170 [INFO]  [multi] job_id 16966568
proc_lsst.multi INFO: [multi] job_id 16966568
2024-03-09 17:36:52 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 33
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33
2024-03-09 17:36:52 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:52 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966568
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966568
parsl.executors.status_handling DEBUG: Launched block 88 on executor multi with job ID 16966568
parsl.executors.status_handling INFO: Allocated block ID 89
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-89
2024-03-09 17:36:52 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-89
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-89
2024-03-09 17:36:52 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:52 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:52 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:52 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:52 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 33
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33
2024-03-09 17:36:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:52 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:52 proc_lsst.multi:170 [INFO]  [multi] job_id 16966569
proc_lsst.multi INFO: [multi] job_id 16966569
2024-03-09 17:36:52 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 34
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34
2024-03-09 17:36:52 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:52 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966569
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966569
parsl.executors.status_handling DEBUG: Launched block 89 on executor multi with job ID 16966569
parsl.executors.status_handling INFO: Allocated block ID 90
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-90
2024-03-09 17:36:52 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-90
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-90
2024-03-09 17:36:52 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:52 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:52 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:52 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:52 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 34
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34
2024-03-09 17:36:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:52 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:52 proc_lsst.multi:170 [INFO]  [multi] job_id 16966570
proc_lsst.multi INFO: [multi] job_id 16966570
2024-03-09 17:36:52 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 35
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35
2024-03-09 17:36:52 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:52 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966570
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966570
parsl.executors.status_handling DEBUG: Launched block 90 on executor multi with job ID 16966570
parsl.executors.status_handling INFO: Allocated block ID 91
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-91
2024-03-09 17:36:52 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-91
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-91
2024-03-09 17:36:52 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:52 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:52 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:52 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:52 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 35
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35
2024-03-09 17:36:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:52 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:54 proc_lsst.multi:170 [INFO]  [multi] job_id 16966571
proc_lsst.multi INFO: [multi] job_id 16966571
2024-03-09 17:36:54 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 36
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36
2024-03-09 17:36:54 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:54 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966571
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966571
parsl.executors.status_handling DEBUG: Launched block 91 on executor multi with job ID 16966571
parsl.executors.status_handling INFO: Allocated block ID 92
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-92
2024-03-09 17:36:54 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-92
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-92
2024-03-09 17:36:54 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-09 17:36:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-09 17:36:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-09 17:36:54 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-09 17:36:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-09 17:36:54 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-09 17:36:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-09 17:36:54 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-09 17:36:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-09 17:36:54 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-09 17:36:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 36
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36
2024-03-09 17:36:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:54 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3010,127.0.0.1,10.64.129.10,198.48.92.26,10.64.65.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54168 --result_port=54702 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-09 17:36:55 proc_lsst.multi:170 [INFO]  [multi] job_id 16966572
proc_lsst.multi INFO: [multi] job_id 16966572
2024-03-09 17:36:55 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 37
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37
2024-03-09 17:36:55 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-09 17:36:55 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 16966572
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16966572
parsl.executors.status_handling DEBUG: Launched block 92 on executor multi with job ID 16966572
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23383474330768
2024-03-09 17:36:55 proc_lsst.multi:146 [INFO]  found job 64340 in provider local
proc_lsst.multi INFO: found job 64340 in provider local
2024-03-09 17:36:55 proc_lsst.multi:146 [INFO]  found job 16966475 in provider astro
proc_lsst.multi INFO: found job 16966475 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:36:56 proc_lsst.multi:146 [INFO]  found job 16966476 in provider astro
proc_lsst.multi INFO: found job 16966476 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:36:56 proc_lsst.multi:146 [INFO]  found job 16966477 in provider astro
proc_lsst.multi INFO: found job 16966477 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:36:57 proc_lsst.multi:146 [INFO]  found job 16966478 in provider astro
proc_lsst.multi INFO: found job 16966478 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:36:57 proc_lsst.multi:146 [INFO]  found job 16966479 in provider astro
proc_lsst.multi INFO: found job 16966479 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:36:58 proc_lsst.multi:146 [INFO]  found job 16966480 in provider astro
proc_lsst.multi INFO: found job 16966480 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a4c3959c8f2321e221ff6d854f288cc7 with result from task 55
parsl.dataflow.dflow DEBUG: Task 55 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/cea155c1-dee8-4d9f-bbd5-7a1f5de92c1d_isrForCrosstalkSources_855716_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/cea155c1-dee8-4d9f-bbd5-7a1f5de92c1d_isrForCrosstalkSources_855716_29.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:36:58 proc_lsst.multi:146 [INFO]  found job 16966481 in provider astro
proc_lsst.multi INFO: found job 16966481 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c949a471d980d2f1bb5c3f6b59c74fe9 with result from task 56
parsl.dataflow.dflow DEBUG: Task 56 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/f1a11b4d-3f3c-413a-80b6-d8211f1d2740_isrForCrosstalkSources_855808_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/f1a11b4d-3f3c-413a-80b6-d8211f1d2740_isrForCrosstalkSources_855808_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:36:59 proc_lsst.multi:146 [INFO]  found job 16966482 in provider astro
proc_lsst.multi INFO: found job 16966482 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:36:59 proc_lsst.multi:146 [INFO]  found job 16966483 in provider astro
proc_lsst.multi INFO: found job 16966483 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:00 proc_lsst.multi:146 [INFO]  found job 16966484 in provider astro
proc_lsst.multi INFO: found job 16966484 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:00 proc_lsst.multi:146 [INFO]  found job 16966485 in provider astro
proc_lsst.multi INFO: found job 16966485 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6cf88871ee6df787f3257bbdb8472bdf with result from task 66
parsl.dataflow.dflow DEBUG: Task 66 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/09729c3b-674c-4d95-b781-7b7788f18cdf_isrForCrosstalkSources_855808_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/09729c3b-674c-4d95-b781-7b7788f18cdf_isrForCrosstalkSources_855808_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:01 proc_lsst.multi:146 [INFO]  found job 16966486 in provider astro
proc_lsst.multi INFO: found job 16966486 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:01 proc_lsst.multi:146 [INFO]  found job 16966487 in provider astro
proc_lsst.multi INFO: found job 16966487 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:02 proc_lsst.multi:146 [INFO]  found job 16966488 in provider astro
proc_lsst.multi INFO: found job 16966488 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:02 proc_lsst.multi:146 [INFO]  found job 16966489 in provider astro
proc_lsst.multi INFO: found job 16966489 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:03 proc_lsst.multi:146 [INFO]  found job 16966490 in provider astro
proc_lsst.multi INFO: found job 16966490 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:04 proc_lsst.multi:146 [INFO]  found job 16966491 in provider astro
proc_lsst.multi INFO: found job 16966491 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:04 proc_lsst.multi:146 [INFO]  found job 16966492 in provider astro
proc_lsst.multi INFO: found job 16966492 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:05 proc_lsst.multi:146 [INFO]  found job 16966493 in provider astro
proc_lsst.multi INFO: found job 16966493 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:05 proc_lsst.multi:146 [INFO]  found job 16966499 in provider astro
proc_lsst.multi INFO: found job 16966499 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:05 proc_lsst.multi:146 [INFO]  found job 16966500 in provider astro
proc_lsst.multi INFO: found job 16966500 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:06 proc_lsst.multi:146 [INFO]  found job 16966501 in provider astro
proc_lsst.multi INFO: found job 16966501 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:06 proc_lsst.multi:146 [INFO]  found job 16966502 in provider astro
proc_lsst.multi INFO: found job 16966502 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5c307fe04b4e44b64db678d6110346ea with result from task 67
parsl.dataflow.dflow DEBUG: Task 67 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/2deda282-7e75-42f6-be0b-cc23347e91d4_isrForCrosstalkSources_855717_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/2deda282-7e75-42f6-be0b-cc23347e91d4_isrForCrosstalkSources_855717_25.stderr
parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 059102b5ecff080400c40d44d538fcb6 with result from task 68
parsl.dataflow.dflow DEBUG: Task 68 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/ad209952-fb69-4ef3-a60b-fe9fcc38e1ce_isrForCrosstalkSources_855718_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/ad209952-fb69-4ef3-a60b-fe9fcc38e1ce_isrForCrosstalkSources_855718_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:06 proc_lsst.multi:146 [INFO]  found job 16966503 in provider astro
proc_lsst.multi INFO: found job 16966503 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:07 proc_lsst.multi:146 [INFO]  found job 16966504 in provider astro
proc_lsst.multi INFO: found job 16966504 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:08 proc_lsst.multi:146 [INFO]  found job 16966505 in provider astro
proc_lsst.multi INFO: found job 16966505 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:08 proc_lsst.multi:146 [INFO]  found job 16966506 in provider astro
proc_lsst.multi INFO: found job 16966506 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:09 proc_lsst.multi:146 [INFO]  found job 16966507 in provider astro
proc_lsst.multi INFO: found job 16966507 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:09 proc_lsst.multi:146 [INFO]  found job 16966508 in provider astro
proc_lsst.multi INFO: found job 16966508 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:10 proc_lsst.multi:146 [INFO]  found job 16966509 in provider astro
proc_lsst.multi INFO: found job 16966509 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6adc5e0f79dc2b3097328d2040a759f5 with result from task 47
parsl.dataflow.dflow DEBUG: Task 47 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/f6cb9742-6ea7-4cfa-97fa-42e84ef748e6_isrForCrosstalkSources_855718_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/f6cb9742-6ea7-4cfa-97fa-42e84ef748e6_isrForCrosstalkSources_855718_38.stderr
parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e210935d0d7cfe11e66a70b7b98880ce with result from task 39
parsl.dataflow.dflow DEBUG: Task 39 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/4612ccef-0d84-4a2a-a549-a89de00e9f7a_isrForCrosstalkSources_855809_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/4612ccef-0d84-4a2a-a549-a89de00e9f7a_isrForCrosstalkSources_855809_16.stderr
parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b6668669183c4f12d9238a34dcb4f461 with result from task 46
parsl.dataflow.dflow DEBUG: Task 46 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/43e9dc8b-c23e-4e90-a53a-d6e8256e4c60_isrForCrosstalkSources_855716_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/43e9dc8b-c23e-4e90-a53a-d6e8256e4c60_isrForCrosstalkSources_855716_8.stderr
parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 829f7dc4c284d3f9dd62e01c24a10c25 with result from task 45
parsl.dataflow.dflow DEBUG: Task 45 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/c3300561-3486-4e63-83c3-adff3488dda4_isrForCrosstalkSources_855718_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/c3300561-3486-4e63-83c3-adff3488dda4_isrForCrosstalkSources_855718_2.stderr
parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 22a41d9d59cbb083db7ba9fdf15fe2ce with result from task 43
parsl.dataflow.dflow DEBUG: Task 43 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/0dbbbf0e-1146-40ea-a750-9b84d9b4b996_isrForCrosstalkSources_855717_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/0dbbbf0e-1146-40ea-a750-9b84d9b4b996_isrForCrosstalkSources_855717_46.stderr
parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f0a223ea268eeadf0a0011bb27e1e5a5 with result from task 40
parsl.dataflow.dflow DEBUG: Task 40 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/dbd57a2f-2a07-4f08-9815-0ceb97d5006b_isrForCrosstalkSources_855718_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/dbd57a2f-2a07-4f08-9815-0ceb97d5006b_isrForCrosstalkSources_855718_30.stderr
parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 061aea2e94e90263965b5b0dfad6c711 with result from task 44
parsl.dataflow.dflow DEBUG: Task 44 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/d2c77852-7ae0-4ff4-aa00-1e8c09b78114_isrForCrosstalkSources_855809_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/d2c77852-7ae0-4ff4-aa00-1e8c09b78114_isrForCrosstalkSources_855809_4.stderr
parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 166dc5c2d38a4231010edaf7c38ebb70 with result from task 41
parsl.dataflow.dflow DEBUG: Task 41 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/166b47c9-fe35-4bef-9bdd-af5f4b0e8166_isrForCrosstalkSources_855809_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/166b47c9-fe35-4bef-9bdd-af5f4b0e8166_isrForCrosstalkSources_855809_3.stderr
parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 395c6022941d26558daed5b819bb59a6 with result from task 48
parsl.dataflow.dflow DEBUG: Task 48 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/cfb36cab-963d-4c52-8b12-11a689fbc642_isrForCrosstalkSources_855810_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/cfb36cab-963d-4c52-8b12-11a689fbc642_isrForCrosstalkSources_855810_53.stderr
parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fabf2d4a28b381a8e78b756d240e75e2 with result from task 42
parsl.dataflow.dflow DEBUG: Task 42 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/776cb1ff-f14d-43bb-9f2d-be0415658aee_isrForCrosstalkSources_855717_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/776cb1ff-f14d-43bb-9f2d-be0415658aee_isrForCrosstalkSources_855717_4.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:10 proc_lsst.multi:146 [INFO]  found job 16966510 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966510 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966510,16966511,16966512,16966513,16966514'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966511 R
16966512 R
16966513 R
16966514 R
16966510 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966511 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966512 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966513 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966514 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966510 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:11 proc_lsst.multi:146 [INFO]  found job 16966511 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966511 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966510,16966511,16966512,16966513,16966514'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966511 R
16966512 R
16966513 R
16966514 R
16966510 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966511 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966512 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966513 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966514 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966510 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:11 proc_lsst.multi:146 [INFO]  found job 16966512 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966512 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966510,16966511,16966512,16966513,16966514'
parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fc8d6737668eecfcffe1c0c04da3b766 with result from task 72
parsl.dataflow.dflow DEBUG: Task 72 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/5a5bb503-e84a-416c-bb4b-90f074ba3e2b_isrForCrosstalkSources_855808_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/5a5bb503-e84a-416c-bb4b-90f074ba3e2b_isrForCrosstalkSources_855808_32.stderr
parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry df7eeb5615ccf5d3ef79b273c30eb666 with result from task 70
parsl.dataflow.dflow DEBUG: Task 70 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/6d228380-6472-446d-a86f-93a11a023617_isrForCrosstalkSources_855717_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/6d228380-6472-446d-a86f-93a11a023617_isrForCrosstalkSources_855717_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966511 R
16966512 R
16966513 R
16966514 R
16966510 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966511 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966512 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966513 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966514 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966510 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:12 proc_lsst.multi:146 [INFO]  found job 16966513 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966513 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966510,16966511,16966512,16966513,16966514'
parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 10b3efcc3935dddabe7e14b6f8469e2a with result from task 71
parsl.dataflow.dflow DEBUG: Task 71 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/40e56478-b601-485c-8b66-150bd5b4f6b0_isrForCrosstalkSources_855716_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/40e56478-b601-485c-8b66-150bd5b4f6b0_isrForCrosstalkSources_855716_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966511 R
16966512 R
16966513 R
16966514 R
16966510 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966511 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966512 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966513 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966514 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966510 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:12 proc_lsst.multi:146 [INFO]  found job 16966514 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966514 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966510,16966511,16966512,16966513,16966514'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966511 R
16966512 R
16966513 R
16966514 R
16966510 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966511 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966512 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966513 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966514 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966510 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:13 proc_lsst.multi:146 [INFO]  found job 16966515 in provider escience_a40
proc_lsst.multi INFO: found job 16966515 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:13 proc_lsst.multi:146 [INFO]  found job 16966516 in provider escience_a40
proc_lsst.multi INFO: found job 16966516 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:14 proc_lsst.multi:146 [INFO]  found job 16966517 in provider escience_a40
proc_lsst.multi INFO: found job 16966517 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:14 proc_lsst.multi:146 [INFO]  found job 16966518 in provider escience_a40
proc_lsst.multi INFO: found job 16966518 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:15 proc_lsst.multi:146 [INFO]  found job 16966519 in provider escience_a40
proc_lsst.multi INFO: found job 16966519 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cd524a4cc41d1cf62741741348ac8c08 with result from task 22
parsl.dataflow.dflow DEBUG: Task 22 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/5a509f48-01da-4cb3-b3b1-604a4e81eac5_isrForCrosstalkSources_855718_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/5a509f48-01da-4cb3-b3b1-604a4e81eac5_isrForCrosstalkSources_855718_27.stderr
parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9a319890af47cd32a1ce510184f0a01a with result from task 33
parsl.dataflow.dflow DEBUG: Task 33 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/e5d316b0-ccb5-4f8e-9fa4-7bbfebca3c0b_isrForCrosstalkSources_855718_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/e5d316b0-ccb5-4f8e-9fa4-7bbfebca3c0b_isrForCrosstalkSources_855718_26.stderr
parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c3ca2a0e11aba690bb4b79d3836f9df4 with result from task 29
parsl.dataflow.dflow DEBUG: Task 29 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/c6bf93ab-79b1-4c90-a770-562cc0dd03ea_isrForCrosstalkSources_855716_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/c6bf93ab-79b1-4c90-a770-562cc0dd03ea_isrForCrosstalkSources_855716_46.stderr
parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2945061cc4a8dbcc786b6b1f8c6ba4fe with result from task 28
parsl.dataflow.dflow DEBUG: Task 28 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/60af3367-db8e-42a6-9fbd-e848665b8351_isrForCrosstalkSources_855809_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/60af3367-db8e-42a6-9fbd-e848665b8351_isrForCrosstalkSources_855809_34.stderr
parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ab11ff9433576da15c633cef45eca83b with result from task 27
parsl.dataflow.dflow DEBUG: Task 27 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/0001b779-175b-4b96-af36-76e7993493e3_isrForCrosstalkSources_855810_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/0001b779-175b-4b96-af36-76e7993493e3_isrForCrosstalkSources_855810_5.stderr
parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry db99705cbc30e01f6874b9d1d62da340 with result from task 16
parsl.dataflow.dflow DEBUG: Task 16 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/aeb2a75a-15f1-4b43-8fa7-9329f3e38608_isrForCrosstalkSources_855810_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/aeb2a75a-15f1-4b43-8fa7-9329f3e38608_isrForCrosstalkSources_855810_39.stderr
parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0dc7a78cad6be00a4e0e82330b5c8706 with result from task 24
parsl.dataflow.dflow DEBUG: Task 24 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/8b47d805-40fb-46b2-bdd8-714e04896ac9_isrForCrosstalkSources_855716_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/8b47d805-40fb-46b2-bdd8-714e04896ac9_isrForCrosstalkSources_855716_38.stderr
parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1c61d1d713669bcfd68a4a68c4c9ff79 with result from task 17
parsl.dataflow.dflow DEBUG: Task 17 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/aca64fdd-b67d-45c8-8714-b49beac981e2_isrForCrosstalkSources_855717_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/aca64fdd-b67d-45c8-8714-b49beac981e2_isrForCrosstalkSources_855717_33.stderr
parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a0cf8a58a200a34925ffcc83e5aef8f8 with result from task 26
parsl.dataflow.dflow DEBUG: Task 26 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/0a01f56b-7905-414b-9dc6-5d7207f69e28_isrForCrosstalkSources_855809_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/0a01f56b-7905-414b-9dc6-5d7207f69e28_isrForCrosstalkSources_855809_26.stderr
parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ae9e96844da8e2090e597cf6e618f274 with result from task 11
parsl.dataflow.dflow DEBUG: Task 11 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/7cf151a2-0853-4760-91ba-1df3eec2cd23_isrForCrosstalkSources_855716_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/7cf151a2-0853-4760-91ba-1df3eec2cd23_isrForCrosstalkSources_855716_3.stderr
parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ff14a604a31af0a605e4b2699aa4b09c with result from task 8
parsl.dataflow.dflow DEBUG: Task 8 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/02a48a2b-4731-4fb3-bb12-ab8371f80286_isrForCrosstalkSources_855717_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/02a48a2b-4731-4fb3-bb12-ab8371f80286_isrForCrosstalkSources_855717_20.stderr
parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7df52f6be0d71b8465830020b8f832e4 with result from task 10
parsl.dataflow.dflow DEBUG: Task 10 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/07da1648-b2e9-4dc0-bc89-a2fd2818fa3c_isrForCrosstalkSources_855717_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/07da1648-b2e9-4dc0-bc89-a2fd2818fa3c_isrForCrosstalkSources_855717_19.stderr
parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ecca87c9feaf994d2d85d388aaf3e6c8 with result from task 25
parsl.dataflow.dflow DEBUG: Task 25 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/3810d204-3b2b-4cfb-aa6b-4525dc99b89a_isrForCrosstalkSources_855809_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/3810d204-3b2b-4cfb-aa6b-4525dc99b89a_isrForCrosstalkSources_855809_20.stderr
parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0a96904054bc795a38b1a2fc324756d7 with result from task 15
parsl.dataflow.dflow DEBUG: Task 15 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/0dab09a1-9088-43ae-b0b9-34f25f99bad7_isrForCrosstalkSources_855716_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/0dab09a1-9088-43ae-b0b9-34f25f99bad7_isrForCrosstalkSources_855716_4.stderr
parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 48dbac8e9a1aa95e5aedd41d5e908724 with result from task 30
parsl.dataflow.dflow DEBUG: Task 30 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/d2a44e05-f80c-4052-b0bb-38a597318dd8_isrForCrosstalkSources_855808_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/d2a44e05-f80c-4052-b0bb-38a597318dd8_isrForCrosstalkSources_855808_46.stderr
parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5387da3f1bd2333a3530e67d17978494 with result from task 13
parsl.dataflow.dflow DEBUG: Task 13 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/222efef2-61a9-4a2e-896e-d33a3d15ded1_isrForCrosstalkSources_855810_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/222efef2-61a9-4a2e-896e-d33a3d15ded1_isrForCrosstalkSources_855810_58.stderr
parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 07a5be2cc7f75a522ccfc29ee70091c8 with result from task 12
parsl.dataflow.dflow DEBUG: Task 12 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/ea4e9acc-7aa4-49ab-836c-c52c68eb5ab1_isrForCrosstalkSources_855809_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/ea4e9acc-7aa4-49ab-836c-c52c68eb5ab1_isrForCrosstalkSources_855809_48.stderr
parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 39eaa6b18f96a31dd413444c3f4ea180 with result from task 9
parsl.dataflow.dflow DEBUG: Task 9 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/a5d27239-3931-42f1-8a31-75a33fbd1878_isrForCrosstalkSources_855716_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/a5d27239-3931-42f1-8a31-75a33fbd1878_isrForCrosstalkSources_855716_6.stderr
parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9af76c7e0bbd6c3cf6c89a4a2333d9b5 with result from task 19
parsl.dataflow.dflow DEBUG: Task 19 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/647245f5-1866-47ec-882c-bce615df20ed_isrForCrosstalkSources_855810_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/647245f5-1866-47ec-882c-bce615df20ed_isrForCrosstalkSources_855810_60.stderr
parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ed75f6c62c179e294aa51513b0aad489 with result from task 14
parsl.dataflow.dflow DEBUG: Task 14 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/e4feae2e-0122-414c-b93a-8e78f52055af_isrForCrosstalkSources_855716_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/e4feae2e-0122-414c-b93a-8e78f52055af_isrForCrosstalkSources_855716_35.stderr
parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2648ddb424a02a5b7d1ff8011f01ce35 with result from task 20
parsl.dataflow.dflow DEBUG: Task 20 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/79681c56-f258-44ef-b592-54d49d2040d8_isrForCrosstalkSources_855808_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/79681c56-f258-44ef-b592-54d49d2040d8_isrForCrosstalkSources_855808_10.stderr
parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 887bd84f374e48aaf51145133aaeab31 with result from task 7
parsl.dataflow.dflow DEBUG: Task 7 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/76e99c69-5cbc-457a-afe0-724c118c0430_isrForCrosstalkSources_855809_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/76e99c69-5cbc-457a-afe0-724c118c0430_isrForCrosstalkSources_855809_37.stderr
parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0c76cd2c1b138e40525d27067ed23393 with result from task 21
parsl.dataflow.dflow DEBUG: Task 21 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/52b5d730-60bb-4866-bb66-e8022b3e852c_isrForCrosstalkSources_855809_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/52b5d730-60bb-4866-bb66-e8022b3e852c_isrForCrosstalkSources_855809_41.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:16 proc_lsst.multi:146 [INFO]  found job 16966520 in provider escience_a40
proc_lsst.multi INFO: found job 16966520 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ee91e50fac6ecab3366c95abd1df8da5 with result from task 32
parsl.dataflow.dflow DEBUG: Task 32 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/5e7914de-62d4-483d-95d3-2689dbab96aa_isrForCrosstalkSources_855809_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/5e7914de-62d4-483d-95d3-2689dbab96aa_isrForCrosstalkSources_855809_47.stderr
parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6ff914cf2737624055d9f09c365b425a with result from task 18
parsl.dataflow.dflow DEBUG: Task 18 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/9f267cbb-16bb-4901-9127-bd1b03be102c_isrForCrosstalkSources_855717_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/9f267cbb-16bb-4901-9127-bd1b03be102c_isrForCrosstalkSources_855717_6.stderr
parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 07622cdfb2e4af7e39f118d1a078d75d with result from task 31
parsl.dataflow.dflow DEBUG: Task 31 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/76b76337-a41e-47e1-971b-8e67509d617d_isrForCrosstalkSources_855808_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/76b76337-a41e-47e1-971b-8e67509d617d_isrForCrosstalkSources_855808_41.stderr
parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 47ae7f6094524d763e7c76cce6096fc8 with result from task 23
parsl.dataflow.dflow DEBUG: Task 23 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/66cd32b9-9387-47f7-9820-f869273dc0a8_isrForCrosstalkSources_855718_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/66cd32b9-9387-47f7-9820-f869273dc0a8_isrForCrosstalkSources_855718_31.stderr
parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2312537624ef7e47924fce07ad70f769 with result from task 63
parsl.dataflow.dflow DEBUG: Task 63 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/435f723a-37a2-4c7f-976e-369415eb756d_isrForCrosstalkSources_855717_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/435f723a-37a2-4c7f-976e-369415eb756d_isrForCrosstalkSources_855717_13.stderr
parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6fea6aeb886e5b362290ab315eafbea5 with result from task 59
parsl.dataflow.dflow DEBUG: Task 59 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/01276b26-4bab-4fa2-a763-adf3a0a7c442_isrForCrosstalkSources_855718_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/01276b26-4bab-4fa2-a763-adf3a0a7c442_isrForCrosstalkSources_855718_56.stderr
parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5e5e33e08459aeff49ae89d33dd83e76 with result from task 65
parsl.dataflow.dflow DEBUG: Task 65 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/ad81d948-66c1-49cf-9a15-45e3605b72d9_isrForCrosstalkSources_855718_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/ad81d948-66c1-49cf-9a15-45e3605b72d9_isrForCrosstalkSources_855718_45.stderr
parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1040372fd60f0e7185038a89b2a92773 with result from task 61
parsl.dataflow.dflow DEBUG: Task 61 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/f7ebd5fe-319e-4434-8471-e2672aeeef86_isrForCrosstalkSources_855717_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/f7ebd5fe-319e-4434-8471-e2672aeeef86_isrForCrosstalkSources_855717_26.stderr
parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c5c02a1b27b918cab49399c1079bc8c8 with result from task 62
parsl.dataflow.dflow DEBUG: Task 62 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/2335573c-64d5-41db-acb3-78adf94d9a87_isrForCrosstalkSources_855810_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/2335573c-64d5-41db-acb3-78adf94d9a87_isrForCrosstalkSources_855810_22.stderr
parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b225a6a3c48300af1da1dcf4b6a603c8 with result from task 57
parsl.dataflow.dflow DEBUG: Task 57 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/da55b8ca-cf85-4480-a561-784dbf1a8d1a_isrForCrosstalkSources_855810_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/da55b8ca-cf85-4480-a561-784dbf1a8d1a_isrForCrosstalkSources_855810_40.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:18 proc_lsst.multi:146 [INFO]  found job 16966521 in provider escience_a40
proc_lsst.multi INFO: found job 16966521 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c432e4052839e641bbef0a17c52860ab with result from task 60
parsl.dataflow.dflow DEBUG: Task 60 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/e4d61c22-0f03-4351-a633-3b3f78fec695_isrForCrosstalkSources_855716_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/e4d61c22-0f03-4351-a633-3b3f78fec695_isrForCrosstalkSources_855716_30.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:18 proc_lsst.multi:146 [INFO]  found job 16966522 in provider escience_a40
proc_lsst.multi INFO: found job 16966522 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0595f31bf42121c15a9b7babaee5b55b with result from task 58
parsl.dataflow.dflow DEBUG: Task 58 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/caac27a7-2331-4ed5-879a-262565331a77_isrForCrosstalkSources_855716_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/caac27a7-2331-4ed5-879a-262565331a77_isrForCrosstalkSources_855716_5.stderr
parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 566dc2d10c03d43c28761e537a52c66f with result from task 64
parsl.dataflow.dflow DEBUG: Task 64 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/7de08bde-143e-43ed-91d6-d42f77e2b1f2_isrForCrosstalkSources_855716_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/7de08bde-143e-43ed-91d6-d42f77e2b1f2_isrForCrosstalkSources_855716_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:19 proc_lsst.multi:146 [INFO]  found job 16966523 in provider escience_a40
proc_lsst.multi INFO: found job 16966523 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:19 proc_lsst.multi:146 [INFO]  found job 16966524 in provider escience_a40
proc_lsst.multi INFO: found job 16966524 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:20 proc_lsst.multi:146 [INFO]  found job 16966525 in provider escience_a40
proc_lsst.multi INFO: found job 16966525 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:20 proc_lsst.multi:146 [INFO]  found job 16966526 in provider escience_a40
proc_lsst.multi INFO: found job 16966526 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:21 proc_lsst.multi:146 [INFO]  found job 16966527 in provider escience_a40
proc_lsst.multi INFO: found job 16966527 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d153b5d46e8fd6bf3c9b56b343a8b6fc with result from task 78
parsl.dataflow.dflow DEBUG: Task 78 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/c9a8ddd0-ff74-4efb-8762-e3aace0cc9b0_isrForCrosstalkSources_855717_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/c9a8ddd0-ff74-4efb-8762-e3aace0cc9b0_isrForCrosstalkSources_855717_32.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:21 proc_lsst.multi:146 [INFO]  found job 16966528 in provider escience_a40
proc_lsst.multi INFO: found job 16966528 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 76af9f007f7a42da191cd25a2a19ff5f with result from task 82
parsl.dataflow.dflow DEBUG: Task 82 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/cea8637a-f1c3-40d6-bf0f-05f1e1d76ae9_isrForCrosstalkSources_855808_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/cea8637a-f1c3-40d6-bf0f-05f1e1d76ae9_isrForCrosstalkSources_855808_4.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:22 proc_lsst.multi:146 [INFO]  found job 16966529 in provider escience_a40
proc_lsst.multi INFO: found job 16966529 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e36a2ebbe76df142af87f29ee2c33432 with result from task 73
parsl.dataflow.dflow DEBUG: Task 73 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/76111870-61cf-4237-a4d1-0d03ae25e7d3_isrForCrosstalkSources_855717_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/76111870-61cf-4237-a4d1-0d03ae25e7d3_isrForCrosstalkSources_855717_18.stderr
parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eacc0aed9cf1e4591edf22539e49dd5f with result from task 84
parsl.dataflow.dflow DEBUG: Task 84 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/d1cae28d-2310-4699-88a5-2d4156e9ecd6_isrForCrosstalkSources_855809_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/d1cae28d-2310-4699-88a5-2d4156e9ecd6_isrForCrosstalkSources_855809_40.stderr
parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 19c2c0ce1cd855119b5d0c8a00ea79f3 with result from task 75
parsl.dataflow.dflow DEBUG: Task 75 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/25e524dd-049b-41ee-86f8-426d6defa23b_isrForCrosstalkSources_855808_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/25e524dd-049b-41ee-86f8-426d6defa23b_isrForCrosstalkSources_855808_15.stderr
parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a110ee4ce913948ffec2d3ef833ff42a with result from task 80
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:22 proc_lsst.multi:146 [INFO]  found job 16966530 in provider escience_a40
proc_lsst.multi INFO: found job 16966530 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.dataflow.dflow DEBUG: Task 80 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/2486b497-b695-4067-b1e8-d527d5b9667f_isrForCrosstalkSources_855718_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/2486b497-b695-4067-b1e8-d527d5b9667f_isrForCrosstalkSources_855718_17.stderr
parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 22f009f292402ff49862d263ca419c3a with result from task 74
parsl.dataflow.dflow DEBUG: Task 74 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/de91162e-9dd1-4a6b-92c9-6897e1ca0e23_isrForCrosstalkSources_855717_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/de91162e-9dd1-4a6b-92c9-6897e1ca0e23_isrForCrosstalkSources_855717_28.stderr
parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4686aeb64f6933a3c38afd6ef3566550 with result from task 50
parsl.dataflow.dflow DEBUG: Task 50 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/198be7e5-42c4-4110-a0c5-217b67e74085_isrForCrosstalkSources_855717_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/198be7e5-42c4-4110-a0c5-217b67e74085_isrForCrosstalkSources_855717_42.stderr
parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 87866e70a763c5a83f3a8a01a1e87a07 with result from task 51
parsl.dataflow.dflow DEBUG: Task 51 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/0653461c-b488-4769-b191-4382b58b75c6_isrForCrosstalkSources_855810_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/0653461c-b488-4769-b191-4382b58b75c6_isrForCrosstalkSources_855810_29.stderr
parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry abffc2addffe1991fdf17b6b104060d7 with result from task 54
parsl.dataflow.dflow DEBUG: Task 54 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/fd2abb0d-82ff-4fa1-b916-e17c1e495bfb_isrForCrosstalkSources_855716_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/fd2abb0d-82ff-4fa1-b916-e17c1e495bfb_isrForCrosstalkSources_855716_54.stderr
parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry beb5e700bc2b38cd01184fc4e77c3268 with result from task 53
parsl.dataflow.dflow DEBUG: Task 53 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/ce5c93c4-c7af-41a3-956a-bebd58fb13ea_isrForCrosstalkSources_855810_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/ce5c93c4-c7af-41a3-956a-bebd58fb13ea_isrForCrosstalkSources_855810_8.stderr
parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eefa566046de88e8da78268d38582f00 with result from task 52
parsl.dataflow.dflow DEBUG: Task 52 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/e23fb8ef-2d6c-41ff-99ab-d0d8d51480e7_isrForCrosstalkSources_855718_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/e23fb8ef-2d6c-41ff-99ab-d0d8d51480e7_isrForCrosstalkSources_855718_46.stderr
parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 99c91126de6bd47352e3aa7c4ae4c4cb with result from task 83
parsl.dataflow.dflow DEBUG: Task 83 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/1942f747-7e79-4c62-b2a3-b6c704e306ba_isrForCrosstalkSources_855718_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/1942f747-7e79-4c62-b2a3-b6c704e306ba_isrForCrosstalkSources_855718_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:23 proc_lsst.multi:146 [INFO]  found job 16966531 in provider escience_a40
proc_lsst.multi INFO: found job 16966531 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:23 proc_lsst.multi:146 [INFO]  found job 16966532 in provider escience_a40
proc_lsst.multi INFO: found job 16966532 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7eca6add3854dad056bcb6fa52f80404 with result from task 77
parsl.dataflow.dflow DEBUG: Task 77 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/387cdca1-cfd5-4c0d-b8f0-c23fb320e13d_isrForCrosstalkSources_855808_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/387cdca1-cfd5-4c0d-b8f0-c23fb320e13d_isrForCrosstalkSources_855808_58.stderr
parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c3b7fbb532ce1f891dc6c847c36d61e7 with result from task 85
parsl.dataflow.dflow DEBUG: Task 85 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/838e2cbf-e2ff-4aa1-8d7f-d5c244e07cb0_isrForCrosstalkSources_855810_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/838e2cbf-e2ff-4aa1-8d7f-d5c244e07cb0_isrForCrosstalkSources_855810_50.stderr
parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c333885b6c6598fd219ee701fc0c48d5 with result from task 81
parsl.dataflow.dflow DEBUG: Task 81 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/dc764e95-5a8d-4cb1-bfa0-6e076f4cb408_isrForCrosstalkSources_855716_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/dc764e95-5a8d-4cb1-bfa0-6e076f4cb408_isrForCrosstalkSources_855716_41.stderr
parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ecf3dd37abeb8bfe7976e8c09e70cfa1 with result from task 79
parsl.dataflow.dflow DEBUG: Task 79 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/639f7933-237e-47d5-bd0c-c94d0effcafb_isrForCrosstalkSources_855716_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/639f7933-237e-47d5-bd0c-c94d0effcafb_isrForCrosstalkSources_855716_53.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:24 proc_lsst.multi:146 [INFO]  found job 16966533 in provider escience_a40
proc_lsst.multi INFO: found job 16966533 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:24 proc_lsst.multi:146 [INFO]  found job 16966534 in provider escience_a40
proc_lsst.multi INFO: found job 16966534 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6556ea6835908cca748df916dc2e4cde with result from task 87
parsl.dataflow.dflow DEBUG: Task 87 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/4726fb06-90fe-4467-a64b-f50672469e9f_isrForCrosstalkSources_855808_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/4726fb06-90fe-4467-a64b-f50672469e9f_isrForCrosstalkSources_855808_8.stderr
parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9d1d2e8e5aa8afc6aa0c73fe1f9f30b4 with result from task 86
parsl.dataflow.dflow DEBUG: Task 86 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/1c325253-24bb-41c2-a794-b054a78cfbc6_isrForCrosstalkSources_855810_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/1c325253-24bb-41c2-a794-b054a78cfbc6_isrForCrosstalkSources_855810_48.stderr
parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 515ccd8423ca647127d5aac0f71ac6ca with result from task 76
parsl.dataflow.dflow DEBUG: Task 76 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/ff17624b-6839-4b8c-bf35-3fb3e5f5e514_isrForCrosstalkSources_855717_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/ff17624b-6839-4b8c-bf35-3fb3e5f5e514_isrForCrosstalkSources_855717_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:25 proc_lsst.multi:146 [INFO]  found job 16966535 in provider checkpoint
proc_lsst.multi INFO: found job 16966535 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:25 proc_lsst.multi:146 [INFO]  found job 16966536 in provider checkpoint
proc_lsst.multi INFO: found job 16966536 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:26 proc_lsst.multi:146 [INFO]  found job 16966537 in provider checkpoint
proc_lsst.multi INFO: found job 16966537 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:26 proc_lsst.multi:146 [INFO]  found job 16966538 in provider checkpoint
proc_lsst.multi INFO: found job 16966538 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fccce59df34e8bb0eb62f7d8263445f9 with result from task 117
parsl.dataflow.dflow DEBUG: Task 117 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/9625ce93-8488-4d44-9237-fe1a1c915874_isrForCrosstalkSources_855716_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/9625ce93-8488-4d44-9237-fe1a1c915874_isrForCrosstalkSources_855716_40.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:27 proc_lsst.multi:146 [INFO]  found job 16966539 in provider checkpoint
proc_lsst.multi INFO: found job 16966539 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 081eda5320d7a4d75f7987854b7e852c with result from task 118
parsl.dataflow.dflow DEBUG: Task 118 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/661895a0-2546-492c-b5a1-2e6e9e86916e_isrForCrosstalkSources_855810_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/661895a0-2546-492c-b5a1-2e6e9e86916e_isrForCrosstalkSources_855810_21.stderr
parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 438c6b9374586a6b6f53de83d0eb7655 with result from task 119
parsl.dataflow.dflow DEBUG: Task 119 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/13183ba2-a4c2-4303-ab87-ad089230dad4_isrForCrosstalkSources_855808_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/13183ba2-a4c2-4303-ab87-ad089230dad4_isrForCrosstalkSources_855808_25.stderr
parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d43ea30d8dacfc20a3f344efff24fd4f with result from task 96
parsl.dataflow.dflow DEBUG: Task 96 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/27255655-219f-47e0-93b5-b7b6f12c132e_isrForCrosstalkSources_855808_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/27255655-219f-47e0-93b5-b7b6f12c132e_isrForCrosstalkSources_855808_56.stderr
parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 55bcf55d96fa48e95dce6e72717f152c with result from task 88
parsl.dataflow.dflow DEBUG: Task 88 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/de2e2f80-52ae-4f9d-a781-db3f6e9a3cc9_isrForCrosstalkSources_855808_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/de2e2f80-52ae-4f9d-a781-db3f6e9a3cc9_isrForCrosstalkSources_855808_39.stderr
parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4fbafee0534b0590114603130d4c7ee9 with result from task 112
parsl.dataflow.dflow DEBUG: Task 112 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/439c89c7-ef3c-4324-857c-3e98a39e4959_isrForCrosstalkSources_855809_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/439c89c7-ef3c-4324-857c-3e98a39e4959_isrForCrosstalkSources_855809_33.stderr
parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2eb7682bae08e258061d3466ada410d2 with result from task 97
parsl.dataflow.dflow DEBUG: Task 97 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/0570181c-3633-4233-8f93-9b9e2aeef8a0_isrForCrosstalkSources_855808_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/0570181c-3633-4233-8f93-9b9e2aeef8a0_isrForCrosstalkSources_855808_27.stderr
parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 430c91b206f5fc3f5cb4b501b2c43099 with result from task 121
parsl.dataflow.dflow DEBUG: Task 121 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/96729785-2aa3-4b32-9c4e-7ac111d64147_isrForCrosstalkSources_855716_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/96729785-2aa3-4b32-9c4e-7ac111d64147_isrForCrosstalkSources_855716_51.stderr
parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a405d1cf95b7967bc19dbf33576cbfea with result from task 92
parsl.dataflow.dflow DEBUG: Task 92 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/890086b7-433b-4b05-833f-1c5175fb41c2_isrForCrosstalkSources_855808_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/890086b7-433b-4b05-833f-1c5175fb41c2_isrForCrosstalkSources_855808_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:27 proc_lsst.multi:146 [INFO]  found job 16966540 in provider checkpoint
proc_lsst.multi INFO: found job 16966540 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 433c6a032f33f12a31b1d22424e22982 with result from task 122
parsl.dataflow.dflow DEBUG: Task 122 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/16652e05-e1fb-42c1-928b-8d01e04dfcf7_isrForCrosstalkSources_855717_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/16652e05-e1fb-42c1-928b-8d01e04dfcf7_isrForCrosstalkSources_855717_38.stderr
parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1f032b02cfd366f00e8711a2b98545ce with result from task 115
parsl.dataflow.dflow DEBUG: Task 115 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/0a172a13-837a-477a-ade1-e4b220c696e1_isrForCrosstalkSources_855808_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/0a172a13-837a-477a-ade1-e4b220c696e1_isrForCrosstalkSources_855808_61.stderr
parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry def58f29d7883db8bfa3d8a72f72b95b with result from task 109
parsl.dataflow.dflow DEBUG: Task 109 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/4032a593-4129-4935-9e8e-75733bba80be_isrForCrosstalkSources_855718_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/4032a593-4129-4935-9e8e-75733bba80be_isrForCrosstalkSources_855718_13.stderr
parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 34a58b4a4e6e930627125c996079c0ec with result from task 116
parsl.dataflow.dflow DEBUG: Task 116 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/19577493-09e9-42e3-a2a5-fbdf0c57cb96_isrForCrosstalkSources_855808_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/19577493-09e9-42e3-a2a5-fbdf0c57cb96_isrForCrosstalkSources_855808_49.stderr
parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b5048795fd4fbbec8f4bfe80360b5d18 with result from task 123
parsl.dataflow.dflow DEBUG: Task 123 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/b4a414a0-7d01-42a0-b74b-06f7d45105de_isrForCrosstalkSources_855718_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/b4a414a0-7d01-42a0-b74b-06f7d45105de_isrForCrosstalkSources_855718_4.stderr
parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4e743309905e25f4c32c626081de6580 with result from task 120
parsl.dataflow.dflow DEBUG: Task 120 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/58068611-5dd0-4205-9817-cd18c9828ab4_isrForCrosstalkSources_855717_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/58068611-5dd0-4205-9817-cd18c9828ab4_isrForCrosstalkSources_855717_50.stderr
parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7b4506d1d95605c2d4ef478f49b72a05 with result from task 113
parsl.dataflow.dflow DEBUG: Task 113 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/7e6304e4-5cf2-4b06-906e-590dc5345a54_isrForCrosstalkSources_855718_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/7e6304e4-5cf2-4b06-906e-590dc5345a54_isrForCrosstalkSources_855718_32.stderr
parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a1055bfa398f7b2af358c8d8c61c701d with result from task 93
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:28 proc_lsst.multi:146 [INFO]  found job 16966541 in provider checkpoint
proc_lsst.multi INFO: found job 16966541 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.dataflow.dflow DEBUG: Task 93 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/3801c6fe-feb9-4603-95ce-73e7fe4801f0_isrForCrosstalkSources_855718_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/3801c6fe-feb9-4603-95ce-73e7fe4801f0_isrForCrosstalkSources_855718_8.stderr
parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b880ef489cb61ac3eb7672587aa110ab with result from task 106
parsl.dataflow.dflow DEBUG: Task 106 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/e5d03eb9-e4c2-4d1f-9a71-1f5f6173d13c_isrForCrosstalkSources_855810_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/e5d03eb9-e4c2-4d1f-9a71-1f5f6173d13c_isrForCrosstalkSources_855810_17.stderr
parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0377336f799a7856f049e0974a649a73 with result from task 108
parsl.dataflow.dflow DEBUG: Task 108 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/fbaaf73e-514b-42da-8c37-e5eda7bf5362_isrForCrosstalkSources_855810_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/fbaaf73e-514b-42da-8c37-e5eda7bf5362_isrForCrosstalkSources_855810_16.stderr
parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 23631e0a3cf4bd57a125871f9eaacc88 with result from task 110
parsl.dataflow.dflow DEBUG: Task 110 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/4c33492c-830e-47c5-a0cd-09ca77def43b_isrForCrosstalkSources_855717_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/4c33492c-830e-47c5-a0cd-09ca77def43b_isrForCrosstalkSources_855717_22.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:28 proc_lsst.multi:146 [INFO]  found job 16966542 in provider checkpoint
proc_lsst.multi INFO: found job 16966542 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0b7ea1e6a09108a254bf821764b6c85a with result from task 90
parsl.dataflow.dflow DEBUG: Task 90 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/932a6caa-514d-4a83-b278-db8c97f8edbf_isrForCrosstalkSources_855809_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/932a6caa-514d-4a83-b278-db8c97f8edbf_isrForCrosstalkSources_855809_22.stderr
parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1a27fb23ff7426ba0b45303a59488391 with result from task 94
parsl.dataflow.dflow DEBUG: Task 94 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/ead72d24-2bb3-4083-bcd0-32b527ad65de_isrForCrosstalkSources_855809_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/ead72d24-2bb3-4083-bcd0-32b527ad65de_isrForCrosstalkSources_855809_23.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:29 proc_lsst.multi:146 [INFO]  found job 16966543 in provider checkpoint
proc_lsst.multi INFO: found job 16966543 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fbde43215834c77d57920ca21f4b2628 with result from task 98
parsl.dataflow.dflow DEBUG: Task 98 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/29d88c3a-8c57-43b4-8672-d4dd115b0492_isrForCrosstalkSources_855810_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/29d88c3a-8c57-43b4-8672-d4dd115b0492_isrForCrosstalkSources_855810_3.stderr
parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 704e5e0a6741a3d9142a56bb21ef3d22 with result from task 99
parsl.dataflow.dflow DEBUG: Task 99 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/a8753765-698c-4842-825f-4c7c6f9b4004_isrForCrosstalkSources_855810_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/a8753765-698c-4842-825f-4c7c6f9b4004_isrForCrosstalkSources_855810_37.stderr
parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9fb2da88c459a3e2beba4e08fb25dcdb with result from task 101
parsl.dataflow.dflow DEBUG: Task 101 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/ab1f7733-867c-4760-9685-58fdb207b728_isrForCrosstalkSources_855718_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/ab1f7733-867c-4760-9685-58fdb207b728_isrForCrosstalkSources_855718_36.stderr
parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 63d0bc9c2f956772a39a2cf1ae61eae1 with result from task 95
parsl.dataflow.dflow DEBUG: Task 95 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/4773f8dd-5944-4eb9-b0e7-d1e9997b945a_isrForCrosstalkSources_855718_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/4773f8dd-5944-4eb9-b0e7-d1e9997b945a_isrForCrosstalkSources_855718_37.stderr
parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9f78e5d45700ed4e1160b02ffe734a5e with result from task 100
parsl.dataflow.dflow DEBUG: Task 100 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/5668ae2e-9224-4832-b0bc-b4f966ce6098_isrForCrosstalkSources_855716_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/5668ae2e-9224-4832-b0bc-b4f966ce6098_isrForCrosstalkSources_855716_24.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:29 proc_lsst.multi:146 [INFO]  found job 16966544 in provider checkpoint
proc_lsst.multi INFO: found job 16966544 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 52b9104bd72ef23b39e9bacb272a8374 with result from task 103
parsl.dataflow.dflow DEBUG: Task 103 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/e35cde94-b5f0-42e0-a32e-9c8b9366cb21_isrForCrosstalkSources_855809_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/e35cde94-b5f0-42e0-a32e-9c8b9366cb21_isrForCrosstalkSources_855809_1.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:30 proc_lsst.multi:146 [INFO]  found job 16966545 in provider checkpoint
proc_lsst.multi INFO: found job 16966545 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a68d45ff6f18de167d564642ed595244 with result from task 107
parsl.dataflow.dflow DEBUG: Task 107 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/bc96b4f1-780e-4900-a398-b16320dc5a40_isrForCrosstalkSources_855808_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/bc96b4f1-780e-4900-a398-b16320dc5a40_isrForCrosstalkSources_855808_38.stderr
parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e142502268f3e07b8d794f3605aa8502 with result from task 111
parsl.dataflow.dflow DEBUG: Task 111 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/68148ff7-0abe-4fb6-91c8-b49fd04d54b3_isrForCrosstalkSources_855809_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/68148ff7-0abe-4fb6-91c8-b49fd04d54b3_isrForCrosstalkSources_855809_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:30 proc_lsst.multi:146 [INFO]  found job 16966546 in provider checkpoint
proc_lsst.multi INFO: found job 16966546 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 24d57b6d822f1076eba2d764c994be60 with result from task 105
parsl.dataflow.dflow DEBUG: Task 105 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/42f118c9-802d-4d16-9eba-36cb926139a4_isrForCrosstalkSources_855809_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/42f118c9-802d-4d16-9eba-36cb926139a4_isrForCrosstalkSources_855809_61.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:31 proc_lsst.multi:146 [INFO]  found job 16966547 in provider checkpoint
proc_lsst.multi INFO: found job 16966547 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4f8d1c12f3868c9a7c1c58e72ed8b464 with result from task 89
parsl.dataflow.dflow DEBUG: Task 89 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/c5f3148e-6736-4936-a5eb-b63255155017_isrForCrosstalkSources_855717_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/c5f3148e-6736-4936-a5eb-b63255155017_isrForCrosstalkSources_855717_48.stderr
parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6dfffc882bfbd622afcdaeaea02700e8 with result from task 102
parsl.dataflow.dflow DEBUG: Task 102 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/5cffef1b-1cdc-446b-b92d-53bce7c7bee4_isrForCrosstalkSources_855717_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/5cffef1b-1cdc-446b-b92d-53bce7c7bee4_isrForCrosstalkSources_855717_58.stderr
parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5779d685f0cbe0df5229cf81fc9d652d with result from task 104
parsl.dataflow.dflow DEBUG: Task 104 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/861c5309-07c7-4aff-82ec-8bbeec69a243_isrForCrosstalkSources_855716_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/861c5309-07c7-4aff-82ec-8bbeec69a243_isrForCrosstalkSources_855716_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:32 proc_lsst.multi:146 [INFO]  found job 16966548 in provider checkpoint
proc_lsst.multi INFO: found job 16966548 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:32 proc_lsst.multi:146 [INFO]  found job 16966549 in provider checkpoint
proc_lsst.multi INFO: found job 16966549 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:32 proc_lsst.multi:146 [INFO]  found job 16966550 in provider checkpoint
proc_lsst.multi INFO: found job 16966550 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:33 proc_lsst.multi:146 [INFO]  found job 16966551 in provider checkpoint
proc_lsst.multi INFO: found job 16966551 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6fb237f908ac5b45985cca2c0847e8d1 with result from task 91
parsl.dataflow.dflow DEBUG: Task 91 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/77405b0e-dfd2-4103-97b0-b72c084879d4_isrForCrosstalkSources_855718_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/77405b0e-dfd2-4103-97b0-b72c084879d4_isrForCrosstalkSources_855718_59.stderr
parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4a56164af847d9f6267133510679acbc with result from task 114
parsl.dataflow.dflow DEBUG: Task 114 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/ba839ae2-f7c1-40f9-9ce5-9da18637aab9_isrForCrosstalkSources_855810_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/ba839ae2-f7c1-40f9-9ce5-9da18637aab9_isrForCrosstalkSources_855810_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:33 proc_lsst.multi:146 [INFO]  found job 16966553 in provider checkpoint
proc_lsst.multi INFO: found job 16966553 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a9c8f525f3b8285ea60d896312ce4f9c with result from task 133
parsl.dataflow.dflow DEBUG: Task 133 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/7936f9e0-a682-4c0e-8e3b-7d430745429e_isrForCrosstalkSources_855808_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/7936f9e0-a682-4c0e-8e3b-7d430745429e_isrForCrosstalkSources_855808_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:34 proc_lsst.multi:146 [INFO]  found job 16966554 in provider checkpoint
proc_lsst.multi INFO: found job 16966554 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5aec14b6fa2c2c58f7493145582fa284 with result from task 132
parsl.dataflow.dflow DEBUG: Task 132 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/92aaaa83-5a7f-4997-a00a-7f67bb0c5e4f_isrForCrosstalkSources_855810_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/92aaaa83-5a7f-4997-a00a-7f67bb0c5e4f_isrForCrosstalkSources_855810_49.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:35 proc_lsst.multi:146 [INFO]  found job 16966555 in provider checkpoint
proc_lsst.multi INFO: found job 16966555 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry da57276457eb570560c6ab872af27dec with result from task 131
parsl.dataflow.dflow DEBUG: Task 131 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/4cbe993e-140f-4030-9ebb-aa7d5e4b2994_isrForCrosstalkSources_855810_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/4cbe993e-140f-4030-9ebb-aa7d5e4b2994_isrForCrosstalkSources_855810_27.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:35 proc_lsst.multi:146 [INFO]  found job 16966556 in provider checkpoint
proc_lsst.multi INFO: found job 16966556 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f7158b2aa2cac36836948cd56cf3e0db with result from task 151
parsl.dataflow.dflow DEBUG: Task 151 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/5ed13e69-7823-499c-a3d6-8e7149317692_isrForCrosstalkSources_855717_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/5ed13e69-7823-499c-a3d6-8e7149317692_isrForCrosstalkSources_855717_36.stderr
parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fdc9e8384c86cb180536c51e9428d5e6 with result from task 146
parsl.dataflow.dflow DEBUG: Task 146 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/e60af1bb-60fb-43b3-82f0-019093f39d95_isrForCrosstalkSources_855716_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/e60af1bb-60fb-43b3-82f0-019093f39d95_isrForCrosstalkSources_855716_49.stderr
parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e3bb14ecde40c9594f7feddf65f092a5 with result from task 147
parsl.dataflow.dflow DEBUG: Task 147 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/c1ad64db-fd5f-4067-82d7-0585c65ee013_isrForCrosstalkSources_855716_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/c1ad64db-fd5f-4067-82d7-0585c65ee013_isrForCrosstalkSources_855716_39.stderr
parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 76b290dd50b4b0667293990c95421543 with result from task 134
parsl.dataflow.dflow DEBUG: Task 134 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/ef62a15e-e513-4077-88ed-a04b7a347f3d_isrForCrosstalkSources_855810_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/ef62a15e-e513-4077-88ed-a04b7a347f3d_isrForCrosstalkSources_855810_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:36 proc_lsst.multi:146 [INFO]  found job 16966557 in provider checkpoint
proc_lsst.multi INFO: found job 16966557 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5e813724ee19cfbda7a6df3f08f91493 with result from task 127
parsl.dataflow.dflow DEBUG: Task 127 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/02039795-e35e-49d3-b6cc-110edb65347e_isrForCrosstalkSources_855718_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/02039795-e35e-49d3-b6cc-110edb65347e_isrForCrosstalkSources_855718_40.stderr
parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dbb0ecccab5e21d272a7aef0dae1e8ca with result from task 135
parsl.dataflow.dflow DEBUG: Task 135 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/70381a29-fdf4-417f-9c55-06f0cb1aef2a_isrForCrosstalkSources_855717_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/70381a29-fdf4-417f-9c55-06f0cb1aef2a_isrForCrosstalkSources_855717_3.stderr
parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2ed64ed7b1715311c05571f19e93d0cb with result from task 129
parsl.dataflow.dflow DEBUG: Task 129 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/0c5c9f28-21b0-4d99-806c-8db222147e56_isrForCrosstalkSources_855810_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/0c5c9f28-21b0-4d99-806c-8db222147e56_isrForCrosstalkSources_855810_51.stderr
parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 98e6281abd7345ff40c1007e49654620 with result from task 159
parsl.dataflow.dflow DEBUG: Task 159 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/e648df51-ea28-4656-9556-2035d40380c8_isrForCrosstalkSources_855718_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/e648df51-ea28-4656-9556-2035d40380c8_isrForCrosstalkSources_855718_10.stderr
parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cdce0c8b4fbce4763f0b159423f7dd50 with result from task 177
parsl.dataflow.dflow DEBUG: Task 177 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/6b683bc7-2c53-42b8-bb8d-3d428e957617_isrForCrosstalkSources_855716_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/6b683bc7-2c53-42b8-bb8d-3d428e957617_isrForCrosstalkSources_855716_42.stderr
parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8ea55f62b82e6676da7399ab9ed4108b with result from task 153
parsl.dataflow.dflow DEBUG: Task 153 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/0522d5f0-65ea-42c6-90a4-3d05abd67d59_isrForCrosstalkSources_855716_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/0522d5f0-65ea-42c6-90a4-3d05abd67d59_isrForCrosstalkSources_855716_62.stderr
parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7785eccff112daf35119655d052abb2c with result from task 180
parsl.dataflow.dflow DEBUG: Task 180 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/178a81dd-c38f-4e6b-b251-c4090a89d09d_isrForCrosstalkSources_855718_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/178a81dd-c38f-4e6b-b251-c4090a89d09d_isrForCrosstalkSources_855718_48.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:36 proc_lsst.multi:146 [INFO]  found job 16966558 in provider checkpoint
proc_lsst.multi INFO: found job 16966558 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cb0a0c84107f30d175320c6987e8e980 with result from task 125
parsl.dataflow.dflow DEBUG: Task 125 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/fad1697e-7e1b-4c67-ae63-70eabb4d5f9a_isrForCrosstalkSources_855808_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/fad1697e-7e1b-4c67-ae63-70eabb4d5f9a_isrForCrosstalkSources_855808_35.stderr
parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d7b4582e072880ad9960dc48c0026b93 with result from task 154
parsl.dataflow.dflow DEBUG: Task 154 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/d7418246-972e-482e-9f80-5a5cf7c1f9ee_isrForCrosstalkSources_855718_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/d7418246-972e-482e-9f80-5a5cf7c1f9ee_isrForCrosstalkSources_855718_53.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:36 proc_lsst.multi:146 [INFO]  found job 16966559 in provider checkpoint
proc_lsst.multi INFO: found job 16966559 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a137ae9b85a5d27a1aab9607409a2ba2 with result from task 152
parsl.dataflow.dflow DEBUG: Task 152 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/af6f50f6-e4b6-4715-b815-59a436bf3ab6_isrForCrosstalkSources_855809_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/af6f50f6-e4b6-4715-b815-59a436bf3ab6_isrForCrosstalkSources_855809_2.stderr
parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e4bdfcaa7dbb3dc36e11fde9f5787156 with result from task 178
parsl.dataflow.dflow DEBUG: Task 178 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/a90e06cb-c3ae-4fdb-ae54-fac83e3d9cf0_isrForCrosstalkSources_855809_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/a90e06cb-c3ae-4fdb-ae54-fac83e3d9cf0_isrForCrosstalkSources_855809_60.stderr
parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0da8c4fd2d8d0c1b54668f6de5948f85 with result from task 181
parsl.dataflow.dflow DEBUG: Task 181 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/fa4fb907-6ce3-4456-932c-c6deaab9190e_isrForCrosstalkSources_855808_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/fa4fb907-6ce3-4456-932c-c6deaab9190e_isrForCrosstalkSources_855808_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:37 proc_lsst.multi:146 [INFO]  found job 16966560 in provider checkpoint
proc_lsst.multi INFO: found job 16966560 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ef15d30c222b04dd773d728538be9acc with result from task 182
parsl.dataflow.dflow DEBUG: Task 182 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/7ed4f035-ea91-4811-bc9b-12d39caefb58_isrForCrosstalkSources_855810_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/7ed4f035-ea91-4811-bc9b-12d39caefb58_isrForCrosstalkSources_855810_35.stderr
parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f69cde095f1a847735e538ed8fc3302f with result from task 136
parsl.dataflow.dflow DEBUG: Task 136 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/54a36252-084f-441f-8bc7-0ebb718f7a35_isrForCrosstalkSources_855718_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/54a36252-084f-441f-8bc7-0ebb718f7a35_isrForCrosstalkSources_855718_47.stderr
parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a5c8a315c14e8d319dfe24a6fdd285b7 with result from task 148
parsl.dataflow.dflow DEBUG: Task 148 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/e519861e-e03b-48d2-8acc-28cb6813a4b5_isrForCrosstalkSources_855809_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/e519861e-e03b-48d2-8acc-28cb6813a4b5_isrForCrosstalkSources_855809_43.stderr
parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c0aeb4d2c2faa877a9e9f9c8c46b4eb6 with result from task 128
parsl.dataflow.dflow DEBUG: Task 128 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/7491b2a2-3853-490f-a983-3f040fff7a85_isrForCrosstalkSources_855809_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/7491b2a2-3853-490f-a983-3f040fff7a85_isrForCrosstalkSources_855809_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:37 proc_lsst.multi:146 [INFO]  found job 16966561 in provider checkpoint
proc_lsst.multi INFO: found job 16966561 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 93ef0cfed59e09af0b06cf019796182a with result from task 124
parsl.dataflow.dflow DEBUG: Task 124 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/e3acbd45-b9e9-46d4-937a-45115e8f3439_isrForCrosstalkSources_855809_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/e3acbd45-b9e9-46d4-937a-45115e8f3439_isrForCrosstalkSources_855809_54.stderr
parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 63accfd11cb6a2e12435ca27c6b08a93 with result from task 145
parsl.dataflow.dflow DEBUG: Task 145 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/7babea6b-be5c-4194-a09a-d13c725006a6_isrForCrosstalkSources_855808_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/7babea6b-be5c-4194-a09a-d13c725006a6_isrForCrosstalkSources_855808_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:37 proc_lsst.multi:146 [INFO]  found job 16966562 in provider checkpoint
proc_lsst.multi INFO: found job 16966562 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:38 proc_lsst.multi:146 [INFO]  found job 16966563 in provider checkpoint
proc_lsst.multi INFO: found job 16966563 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:38 proc_lsst.multi:146 [INFO]  found job 16966564 in provider checkpoint
proc_lsst.multi INFO: found job 16966564 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:38 proc_lsst.multi:146 [INFO]  found job 16966565 in provider checkpoint
proc_lsst.multi INFO: found job 16966565 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:38 proc_lsst.multi:146 [INFO]  found job 16966566 in provider checkpoint
proc_lsst.multi INFO: found job 16966566 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:39 proc_lsst.multi:146 [INFO]  found job 16966567 in provider checkpoint
proc_lsst.multi INFO: found job 16966567 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:39 proc_lsst.multi:146 [INFO]  found job 16966568 in provider checkpoint
proc_lsst.multi INFO: found job 16966568 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:39 proc_lsst.multi:146 [INFO]  found job 16966569 in provider checkpoint
proc_lsst.multi INFO: found job 16966569 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:39 proc_lsst.multi:146 [INFO]  found job 16966570 in provider checkpoint
proc_lsst.multi INFO: found job 16966570 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:40 proc_lsst.multi:146 [INFO]  found job 16966571 in provider checkpoint
proc_lsst.multi INFO: found job 16966571 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:40 proc_lsst.multi:146 [INFO]  found job 16966572 in provider checkpoint
proc_lsst.multi INFO: found job 16966572 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 54782de9b718424e07cc4b0703116af8 with result from task 149
parsl.dataflow.dflow DEBUG: Task 149 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/38d7fc6e-4ece-4ad8-b1d3-880e22d84cb4_isrForCrosstalkSources_855718_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/38d7fc6e-4ece-4ad8-b1d3-880e22d84cb4_isrForCrosstalkSources_855718_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 223
parsl.dataflow.strategy DEBUG: Executor multi has 223 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23383474330768
2024-03-09 17:37:40 proc_lsst.multi:146 [INFO]  found job 64340 in provider local
proc_lsst.multi INFO: found job 64340 in provider local
2024-03-09 17:37:40 proc_lsst.multi:146 [INFO]  found job 16966475 in provider astro
proc_lsst.multi INFO: found job 16966475 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f819d96da9143bb8b078c55c0e8eadaf with result from task 126
parsl.dataflow.dflow DEBUG: Task 126 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/6c156c9d-50fd-438c-9c65-bb9d4783169d_isrForCrosstalkSources_855717_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/6c156c9d-50fd-438c-9c65-bb9d4783169d_isrForCrosstalkSources_855717_55.stderr
parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d50136b6064257b03b39afa0147dc24b with result from task 130
parsl.dataflow.dflow DEBUG: Task 130 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/fe8eeecf-4769-477b-942a-6b4e2c14af0f_isrForCrosstalkSources_855718_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/fe8eeecf-4769-477b-942a-6b4e2c14af0f_isrForCrosstalkSources_855718_14.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:40 proc_lsst.multi:146 [INFO]  found job 16966476 in provider astro
proc_lsst.multi INFO: found job 16966476 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:41 proc_lsst.multi:146 [INFO]  found job 16966477 in provider astro
proc_lsst.multi INFO: found job 16966477 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:41 proc_lsst.multi:146 [INFO]  found job 16966478 in provider astro
proc_lsst.multi INFO: found job 16966478 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:41 proc_lsst.multi:146 [INFO]  found job 16966479 in provider astro
proc_lsst.multi INFO: found job 16966479 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:41 proc_lsst.multi:146 [INFO]  found job 16966480 in provider astro
proc_lsst.multi INFO: found job 16966480 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a763c561be28146cebbbc43ddac4efcb with result from task 150
parsl.dataflow.dflow DEBUG: Task 150 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/069099a5-4bd1-41e7-a34e-41d99959dc4e_isrForCrosstalkSources_855718_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/069099a5-4bd1-41e7-a34e-41d99959dc4e_isrForCrosstalkSources_855718_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:42 proc_lsst.multi:146 [INFO]  found job 16966481 in provider astro
proc_lsst.multi INFO: found job 16966481 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:42 proc_lsst.multi:146 [INFO]  found job 16966482 in provider astro
proc_lsst.multi INFO: found job 16966482 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:42 proc_lsst.multi:146 [INFO]  found job 16966483 in provider astro
proc_lsst.multi INFO: found job 16966483 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:42 proc_lsst.multi:146 [INFO]  found job 16966484 in provider astro
proc_lsst.multi INFO: found job 16966484 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:43 proc_lsst.multi:146 [INFO]  found job 16966485 in provider astro
proc_lsst.multi INFO: found job 16966485 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:43 proc_lsst.multi:146 [INFO]  found job 16966486 in provider astro
proc_lsst.multi INFO: found job 16966486 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:43 proc_lsst.multi:146 [INFO]  found job 16966487 in provider astro
proc_lsst.multi INFO: found job 16966487 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:43 proc_lsst.multi:146 [INFO]  found job 16966488 in provider astro
proc_lsst.multi INFO: found job 16966488 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:44 proc_lsst.multi:146 [INFO]  found job 16966489 in provider astro
proc_lsst.multi INFO: found job 16966489 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:44 proc_lsst.multi:146 [INFO]  found job 16966490 in provider astro
proc_lsst.multi INFO: found job 16966490 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:44 proc_lsst.multi:146 [INFO]  found job 16966491 in provider astro
proc_lsst.multi INFO: found job 16966491 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:44 proc_lsst.multi:146 [INFO]  found job 16966492 in provider astro
proc_lsst.multi INFO: found job 16966492 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7a46106ab8eba84f32e38ffc2f87649d with result from task 260
parsl.dataflow.dflow DEBUG: Task 260 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/868cbe03-9019-4c8d-980a-4071c2df3125_isrForCrosstalkSources_855810_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/868cbe03-9019-4c8d-980a-4071c2df3125_isrForCrosstalkSources_855810_43.stderr
parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a9a2d1ca1feaca53d5cfdeac35373b71 with result from task 261
parsl.dataflow.dflow DEBUG: Task 261 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/3135be6f-15a6-4ff2-9a47-4bbd303ebe14_isrForCrosstalkSources_855717_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/3135be6f-15a6-4ff2-9a47-4bbd303ebe14_isrForCrosstalkSources_855717_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:45 proc_lsst.multi:146 [INFO]  found job 16966493 in provider astro
proc_lsst.multi INFO: found job 16966493 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:45 proc_lsst.multi:146 [INFO]  found job 16966499 in provider astro
proc_lsst.multi INFO: found job 16966499 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:45 proc_lsst.multi:146 [INFO]  found job 16966500 in provider astro
proc_lsst.multi INFO: found job 16966500 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:46 proc_lsst.multi:146 [INFO]  found job 16966501 in provider astro
proc_lsst.multi INFO: found job 16966501 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:46 proc_lsst.multi:146 [INFO]  found job 16966502 in provider astro
proc_lsst.multi INFO: found job 16966502 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:46 proc_lsst.multi:146 [INFO]  found job 16966503 in provider astro
proc_lsst.multi INFO: found job 16966503 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 679d9bbff69d36a9347fd33804fc3ea1 with result from task 356
parsl.dataflow.dflow DEBUG: Task 356 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/c3d578d8-ae02-41a7-8cd5-83cb769e6a5a_isrForCrosstalkSources_855809_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/c3d578d8-ae02-41a7-8cd5-83cb769e6a5a_isrForCrosstalkSources_855809_38.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:46 proc_lsst.multi:146 [INFO]  found job 16966504 in provider astro
proc_lsst.multi INFO: found job 16966504 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fb50e1e84cc2aa7468c9b9500b1406c2 with result from task 359
parsl.dataflow.dflow DEBUG: Task 359 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/c31939ee-3aae-4e61-a4d2-35186c973490_isrForCrosstalkSources_855718_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/c31939ee-3aae-4e61-a4d2-35186c973490_isrForCrosstalkSources_855718_34.stderr
parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 41e80ddd0337d3d71c39deae0da4c3f6 with result from task 357
parsl.dataflow.dflow DEBUG: Task 357 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/8078361a-58cd-4e1e-8310-29baa4bc8eb6_isrForCrosstalkSources_855718_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/8078361a-58cd-4e1e-8310-29baa4bc8eb6_isrForCrosstalkSources_855718_54.stderr
parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2277f8fa1625425da15c333d7000d953 with result from task 358
parsl.dataflow.dflow DEBUG: Task 358 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/1efa9402-4e53-4d5b-985c-388ae5f5b4e7_isrForCrosstalkSources_855809_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/1efa9402-4e53-4d5b-985c-388ae5f5b4e7_isrForCrosstalkSources_855809_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:46 proc_lsst.multi:146 [INFO]  found job 16966505 in provider astro
proc_lsst.multi INFO: found job 16966505 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:47 proc_lsst.multi:146 [INFO]  found job 16966506 in provider astro
proc_lsst.multi INFO: found job 16966506 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aa6a627513455ba05624fb693c3bac9d with result from task 343
parsl.dataflow.dflow DEBUG: Task 343 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/692ecc34-a6ff-40cd-9eb9-25a2b86560df_isrForCrosstalkSources_855808_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/692ecc34-a6ff-40cd-9eb9-25a2b86560df_isrForCrosstalkSources_855808_31.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:47 proc_lsst.multi:146 [INFO]  found job 16966507 in provider astro
proc_lsst.multi INFO: found job 16966507 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d0d8fcbd6358dcd8300530fa4fcace90 with result from task 225
parsl.dataflow.dflow DEBUG: Task 225 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/31c9cb45-e8b6-4e36-acd6-cf11b8202314_isrForCrosstalkSources_855718_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/31c9cb45-e8b6-4e36-acd6-cf11b8202314_isrForCrosstalkSources_855718_20.stderr
parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f5a626b682509723edfc83fce582d1fe with result from task 345
parsl.dataflow.dflow DEBUG: Task 345 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/0bf5ea5c-9a7a-4e5c-bfb6-e733d5016499_isrForCrosstalkSources_855808_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/0bf5ea5c-9a7a-4e5c-bfb6-e733d5016499_isrForCrosstalkSources_855808_7.stderr
parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 323a4c70bc3e59589cf500fd6ae12322 with result from task 251
parsl.dataflow.dflow DEBUG: Task 251 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/2ebeaeb1-7efe-429c-9b5d-80779d35b85d_isrForCrosstalkSources_855716_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/2ebeaeb1-7efe-429c-9b5d-80779d35b85d_isrForCrosstalkSources_855716_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done)
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.memoization DEBUG: Storing app cache entry f6fec0a91805f8b0253544ad83faa0b0 with result from task 342
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:47 proc_lsst.multi:146 [INFO]  found job 16966508 in provider astro
proc_lsst.multi INFO: found job 16966508 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.dataflow.dflow DEBUG: Task 342 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/af156947-93ce-4c74-a210-22d3a49dbb91_isrForCrosstalkSources_855717_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/af156947-93ce-4c74-a210-22d3a49dbb91_isrForCrosstalkSources_855717_52.stderr
parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f5e3f51049c9ae090ac6adc358970659 with result from task 355
parsl.dataflow.dflow DEBUG: Task 355 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/0ae64c8a-e299-4648-a30b-1cb5ec3c357f_isrForCrosstalkSources_855718_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/0ae64c8a-e299-4648-a30b-1cb5ec3c357f_isrForCrosstalkSources_855718_44.stderr
parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cbd3b764b6612de31ca7d5f1bcbde62f with result from task 344
parsl.dataflow.dflow DEBUG: Task 344 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/8c0ffa0f-4555-452f-a130-c29952cdedd0_isrForCrosstalkSources_855717_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/8c0ffa0f-4555-452f-a130-c29952cdedd0_isrForCrosstalkSources_855717_53.stderr
parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8fc1a699f0d9bf639f6ca6cf7fdab1ad with result from task 250
parsl.dataflow.dflow DEBUG: Task 250 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/9743ab27-5fa3-4424-bf77-89fd4c19cc00_isrForCrosstalkSources_855809_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/9743ab27-5fa3-4424-bf77-89fd4c19cc00_isrForCrosstalkSources_855809_45.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:47 proc_lsst.multi:146 [INFO]  found job 16966509 in provider astro
proc_lsst.multi INFO: found job 16966509 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:48 proc_lsst.multi:146 [INFO]  found job 16966510 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966510 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966510,16966511,16966512,16966513,16966514'
parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1bd93d8cb9787162de16ffade4616913 with result from task 202
parsl.dataflow.dflow DEBUG: Task 202 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/c8799da8-3cd6-4a16-bbb5-6799ff868388_isrForCrosstalkSources_855809_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/c8799da8-3cd6-4a16-bbb5-6799ff868388_isrForCrosstalkSources_855809_46.stderr
parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0e65dc8c3ebcfe74977bbb7ab3be3ca8 with result from task 187
parsl.dataflow.dflow DEBUG: Task 187 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/7d03ab40-ed2d-4054-9de4-09670b728956_isrForCrosstalkSources_855808_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/7d03ab40-ed2d-4054-9de4-09670b728956_isrForCrosstalkSources_855808_21.stderr
parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d37c10faa0b6c0451b6dcbf18d1669cf with result from task 273
parsl.dataflow.dflow DEBUG: Task 273 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/c6c11a86-657e-4ab0-8992-f3bd0dfb4045_isrForCrosstalkSources_855718_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/c6c11a86-657e-4ab0-8992-f3bd0dfb4045_isrForCrosstalkSources_855718_1.stderr
parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 360b6f9c8872a057148060f77ebcdeaf with result from task 213
parsl.dataflow.dflow DEBUG: Task 213 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/5b33f8a7-a8ce-489b-b51c-0755aed1b7ab_isrForCrosstalkSources_855808_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/5b33f8a7-a8ce-489b-b51c-0755aed1b7ab_isrForCrosstalkSources_855808_45.stderr
parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 67469de6bf6bb38788bc815630d921d1 with result from task 179
parsl.dataflow.dflow DEBUG: Task 179 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/c504d0bf-5165-46ba-aff2-1d0919ad5df4_isrForCrosstalkSources_855808_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/c504d0bf-5165-46ba-aff2-1d0919ad5df4_isrForCrosstalkSources_855808_23.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966511 R
16966512 R
16966513 R
16966514 R
16966510 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966511 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966512 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966513 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966514 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966510 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:48 proc_lsst.multi:146 [INFO]  found job 16966511 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966511 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966510,16966511,16966512,16966513,16966514'
parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 58da19817636469e0bb4856a1da16fe8 with result from task 354
parsl.dataflow.dflow DEBUG: Task 354 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/13b5fb80-8934-47a0-afbd-99d5e2bfa2b4_isrForCrosstalkSources_855716_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/13b5fb80-8934-47a0-afbd-99d5e2bfa2b4_isrForCrosstalkSources_855716_33.stderr
parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b1e3b01cd17977ff84ea92bb64a5e58b with result from task 323
parsl.dataflow.dflow DEBUG: Task 323 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/7a082fa1-9173-4249-a726-12bab62a6c9d_isrForCrosstalkSources_855716_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/7a082fa1-9173-4249-a726-12bab62a6c9d_isrForCrosstalkSources_855716_32.stderr
parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 80511578bcfe1d9828cc22aac93b18f1 with result from task 191
parsl.dataflow.dflow DEBUG: Task 191 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/a026161d-2297-48e6-81f9-f39b24cce4cf_isrForCrosstalkSources_855810_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/a026161d-2297-48e6-81f9-f39b24cce4cf_isrForCrosstalkSources_855810_15.stderr
parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8d64b18f185b6e88a55f6caaa6960855 with result from task 223
parsl.dataflow.dflow DEBUG: Task 223 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/efa72da9-6cd2-4959-863c-5f4da95e39ca_isrForCrosstalkSources_855810_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/efa72da9-6cd2-4959-863c-5f4da95e39ca_isrForCrosstalkSources_855810_9.stderr
parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f34dc1199e3b0b52bcbcc712a7c30a10 with result from task 192
parsl.dataflow.dflow DEBUG: Task 192 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/5b29b87d-d67b-4f83-8cc9-f8192a0c37fe_isrForCrosstalkSources_855718_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/5b29b87d-d67b-4f83-8cc9-f8192a0c37fe_isrForCrosstalkSources_855718_3.stderr
parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 362eebf85d03198f08d6df846900d656 with result from task 155
parsl.dataflow.dflow DEBUG: Task 155 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/003d6ff9-a934-4726-8b9a-57d08841626a_isrForCrosstalkSources_855718_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/003d6ff9-a934-4726-8b9a-57d08841626a_isrForCrosstalkSources_855718_7.stderr
parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f4ae18ef2e540fb8875e68c10e130116 with result from task 160
parsl.dataflow.dflow DEBUG: Task 160 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/361350d8-ee74-4901-8707-e04257e00b9f_isrForCrosstalkSources_855717_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/361350d8-ee74-4901-8707-e04257e00b9f_isrForCrosstalkSources_855717_39.stderr
parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 04e6b4b5544a30321a5cef938708ae64 with result from task 333
parsl.dataflow.dflow DEBUG: Task 333 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/cd7a2ecd-26f5-4c8d-b35e-b1bde14f62ab_isrForCrosstalkSources_855810_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/cd7a2ecd-26f5-4c8d-b35e-b1bde14f62ab_isrForCrosstalkSources_855810_56.stderr
parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6911b57bb7e575f7dff4dd136c42d78f with result from task 272
parsl.dataflow.dflow DEBUG: Task 272 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/a439fccd-75d0-4f74-99fa-624672438a2d_isrForCrosstalkSources_855718_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/a439fccd-75d0-4f74-99fa-624672438a2d_isrForCrosstalkSources_855718_39.stderr
parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 521c7e11d4ba57ad61b71faae2d41f51 with result from task 158
parsl.dataflow.dflow DEBUG: Task 158 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/5f3d79e2-4cea-419a-9217-e7406d8a5ec7_isrForCrosstalkSources_855718_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/5f3d79e2-4cea-419a-9217-e7406d8a5ec7_isrForCrosstalkSources_855718_9.stderr
parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8154097830c9cfdd12c4783c6289d900 with result from task 370
parsl.dataflow.dflow DEBUG: Task 370 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/35d3ccbe-d38e-4103-b8ad-fe189d5577ce_isrForCrosstalkSources_855809_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/35d3ccbe-d38e-4103-b8ad-fe189d5577ce_isrForCrosstalkSources_855809_24.stderr
parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d7f6f3b6b092653a69434ff0bfff5dd0 with result from task 157
parsl.dataflow.dflow DEBUG: Task 157 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/60593ba5-9a00-44d3-8e8c-5d0c4ad8dcfd_isrForCrosstalkSources_855810_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/60593ba5-9a00-44d3-8e8c-5d0c4ad8dcfd_isrForCrosstalkSources_855810_2.stderr
parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry faa57f50b2c69ce662d003e0be512b06 with result from task 212
parsl.dataflow.dflow DEBUG: Task 212 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/393d6ce4-206e-4fd9-9c03-6495c59efdc2_isrForCrosstalkSources_855718_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/393d6ce4-206e-4fd9-9c03-6495c59efdc2_isrForCrosstalkSources_855718_43.stderr
parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f8777c80893c9bba6d48dbf7d48a24c8 with result from task 270
parsl.dataflow.dflow DEBUG: Task 270 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/f7add256-d6d4-462c-97c1-647c7d178c55_isrForCrosstalkSources_855808_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/f7add256-d6d4-462c-97c1-647c7d178c55_isrForCrosstalkSources_855808_29.stderr
parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 471aa665b51f8242916ad027769aa59a with result from task 368
parsl.dataflow.dflow DEBUG: Task 368 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/a15ce514-15f0-456f-a38d-e24ae0eb3c4b_isrForCrosstalkSources_855810_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/a15ce514-15f0-456f-a38d-e24ae0eb3c4b_isrForCrosstalkSources_855810_10.stderr
parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f62f2bb5d4f536655bd45bf2a1f43e3f with result from task 193
parsl.dataflow.dflow DEBUG: Task 193 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/e31dffa0-a6bc-41c9-a4c0-d63d52d1c5ac_isrForCrosstalkSources_855810_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/e31dffa0-a6bc-41c9-a4c0-d63d52d1c5ac_isrForCrosstalkSources_855810_28.stderr
parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry db54efaf9a2de9c409de5f7a9ea74c0b with result from task 369
parsl.dataflow.dflow DEBUG: Task 369 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/48ed2d7d-f6a6-405a-88f5-98b4b2d2550d_isrForCrosstalkSources_855809_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/48ed2d7d-f6a6-405a-88f5-98b4b2d2550d_isrForCrosstalkSources_855809_12.stderr
parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bebfbf96db2bc3a10dc3f13ff069a4b4 with result from task 185
parsl.dataflow.dflow DEBUG: Task 185 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/65ff7485-d7c9-414e-a4fb-44696a46c982_isrForCrosstalkSources_855809_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/65ff7485-d7c9-414e-a4fb-44696a46c982_isrForCrosstalkSources_855809_11.stderr
parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e0812e908b155464d95b32459014f768 with result from task 189
parsl.dataflow.dflow DEBUG: Task 189 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/2080af5f-882e-4a6d-890e-2856bee360b0_isrForCrosstalkSources_855716_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/2080af5f-882e-4a6d-890e-2856bee360b0_isrForCrosstalkSources_855716_23.stderr
parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4c22e600c8312961e2c5ae0f9dda33c5 with result from task 183
parsl.dataflow.dflow DEBUG: Task 183 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/eccd42c5-acaa-47f7-84f9-c73289cdcc16_isrForCrosstalkSources_855716_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/eccd42c5-acaa-47f7-84f9-c73289cdcc16_isrForCrosstalkSources_855716_22.stderr
parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e6dfb9d4a586c59a8ae5555e20bdbfae with result from task 271
parsl.dataflow.dflow DEBUG: Task 271 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/c539ea85-1fe7-43a4-9703-4cfe7f3b60a3_isrForCrosstalkSources_855808_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/c539ea85-1fe7-43a4-9703-4cfe7f3b60a3_isrForCrosstalkSources_855808_54.stderr
parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6ab1a51e968639b0e70f4bed2a173105 with result from task 282
parsl.dataflow.dflow DEBUG: Task 282 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/50a17c2c-677b-469b-b664-57966ae1c92a_isrForCrosstalkSources_855718_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/50a17c2c-677b-469b-b664-57966ae1c92a_isrForCrosstalkSources_855718_29.stderr
parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9d7ddc1051e3099040f485fec307304e with result from task 156
parsl.dataflow.dflow DEBUG: Task 156 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/e1b63be9-a798-41d0-8eab-b88f3f5ed4ee_isrForCrosstalkSources_855809_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/e1b63be9-a798-41d0-8eab-b88f3f5ed4ee_isrForCrosstalkSources_855809_31.stderr
parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d13cbd5f6ff9f913eb71b3cd2dd82cce with result from task 371
parsl.dataflow.dflow DEBUG: Task 371 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/7bdba7bc-7091-4bb8-947b-0abdc132ba19_isrForCrosstalkSources_855810_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/7bdba7bc-7091-4bb8-947b-0abdc132ba19_isrForCrosstalkSources_855810_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966511 R
16966512 R
16966513 R
16966514 R
16966510 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966511 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966512 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966513 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966514 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966510 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:50 proc_lsst.multi:146 [INFO]  found job 16966512 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966512 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966510,16966511,16966512,16966513,16966514'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966511 R
16966512 R
16966513 R
16966514 R
16966510 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966511 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966512 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966513 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966514 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966510 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:50 proc_lsst.multi:146 [INFO]  found job 16966513 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966513 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966510,16966511,16966512,16966513,16966514'
parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4ce39d92b292ec4b48357435104bef8b with result from task 222
parsl.dataflow.dflow DEBUG: Task 222 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/ac49009b-fd4b-4826-b880-6d3d7c263368_isrForCrosstalkSources_855717_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/ac49009b-fd4b-4826-b880-6d3d7c263368_isrForCrosstalkSources_855717_41.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966511 R
16966512 R
16966513 R
16966514 R
16966510 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966511 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966512 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966513 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966514 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966510 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:50 proc_lsst.multi:146 [INFO]  found job 16966514 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966514 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966510,16966511,16966512,16966513,16966514'
parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a030b89ce434b7db60ab72dcfd75d9fd with result from task 324
parsl.dataflow.dflow DEBUG: Task 324 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/d4d7f8ca-9750-4ca9-9a31-1444b0897186_isrForCrosstalkSources_855717_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/d4d7f8ca-9750-4ca9-9a31-1444b0897186_isrForCrosstalkSources_855717_61.stderr
parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cb92ef0182d9dfc289662eecb1469b20 with result from task 211
parsl.dataflow.dflow DEBUG: Task 211 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/941a7d7d-ee05-44d6-aafc-0aa32f40d5c0_isrForCrosstalkSources_855808_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/941a7d7d-ee05-44d6-aafc-0aa32f40d5c0_isrForCrosstalkSources_855808_48.stderr
parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8c78d1333752080a6f0e8fd3f7d5197d with result from task 184
parsl.dataflow.dflow DEBUG: Task 184 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/5e1baa19-8b47-4d03-8a4a-8003677ca3da_isrForCrosstalkSources_855810_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/5e1baa19-8b47-4d03-8a4a-8003677ca3da_isrForCrosstalkSources_855810_59.stderr
parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b431d8f7878b4ec3f9defa295e18db13 with result from task 188
parsl.dataflow.dflow DEBUG: Task 188 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/dbb3e4c7-bae6-4736-930b-b80a686c9a35_isrForCrosstalkSources_855808_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/dbb3e4c7-bae6-4736-930b-b80a686c9a35_isrForCrosstalkSources_855808_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966511 R
16966512 R
16966513 R
16966514 R
16966510 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966511 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966512 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966513 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966514 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966510 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:51 proc_lsst.multi:146 [INFO]  found job 16966515 in provider escience_a40
proc_lsst.multi INFO: found job 16966515 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9bede29c71f2b5865a899ce6b7ed7e72 with result from task 190
parsl.dataflow.dflow DEBUG: Task 190 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/b48aaa06-fcd5-41b5-a2e6-5aaecda8de4f_isrForCrosstalkSources_855718_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/b48aaa06-fcd5-41b5-a2e6-5aaecda8de4f_isrForCrosstalkSources_855718_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:51 proc_lsst.multi:146 [INFO]  found job 16966516 in provider escience_a40
proc_lsst.multi INFO: found job 16966516 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b835d2b75d071fee32cf933d9ca419d0 with result from task 224
parsl.dataflow.dflow DEBUG: Task 224 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/102f6582-fd94-4c8c-8709-a958020e786a_isrForCrosstalkSources_855717_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/102f6582-fd94-4c8c-8709-a958020e786a_isrForCrosstalkSources_855717_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:51 proc_lsst.multi:146 [INFO]  found job 16966517 in provider escience_a40
proc_lsst.multi INFO: found job 16966517 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b0869e8fa47f97149e53fabc370427ed with result from task 138
parsl.dataflow.dflow DEBUG: Task 138 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/6fd02e9c-e9f0-4f0f-aaf3-229f004e7ab1_isrForCrosstalkSources_855718_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/6fd02e9c-e9f0-4f0f-aaf3-229f004e7ab1_isrForCrosstalkSources_855718_16.stderr
parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 233bf38ba5b6cfe2ae397db18b6aeeff with result from task 139
parsl.dataflow.dflow DEBUG: Task 139 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/07b61f58-1270-43fd-8f77-c87ad54beb5e_isrForCrosstalkSources_855808_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/07b61f58-1270-43fd-8f77-c87ad54beb5e_isrForCrosstalkSources_855808_20.stderr
parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 88de254df956f0f5213d266c4e92ad23 with result from task 143
parsl.dataflow.dflow DEBUG: Task 143 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/f17410d8-1f77-44ab-8591-4d48725d35cb_isrForCrosstalkSources_855808_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/f17410d8-1f77-44ab-8591-4d48725d35cb_isrForCrosstalkSources_855808_62.stderr
parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8fc70fcac76d009ed1834d68aecfd750 with result from task 186
parsl.dataflow.dflow DEBUG: Task 186 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/135bda14-ad3e-4f00-8f04-267c02ae2c0e_isrForCrosstalkSources_855716_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/135bda14-ad3e-4f00-8f04-267c02ae2c0e_isrForCrosstalkSources_855716_59.stderr
parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 91b7b0aea764e8ac0ff4c7840bab59b1 with result from task 137
parsl.dataflow.dflow DEBUG: Task 137 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/87587180-3ff6-447c-8e6c-432a34d74741_isrForCrosstalkSources_855808_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/87587180-3ff6-447c-8e6c-432a34d74741_isrForCrosstalkSources_855808_19.stderr
parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d2cc75d0c2517f1551125258a30bf47b with result from task 141
parsl.dataflow.dflow DEBUG: Task 141 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/0b291b5c-9f18-4bf8-a2b1-122cbcfd9b8a_isrForCrosstalkSources_855808_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/0b291b5c-9f18-4bf8-a2b1-122cbcfd9b8a_isrForCrosstalkSources_855808_22.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:51 proc_lsst.multi:146 [INFO]  found job 16966518 in provider escience_a40
proc_lsst.multi INFO: found job 16966518 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:52 proc_lsst.multi:146 [INFO]  found job 16966519 in provider escience_a40
proc_lsst.multi INFO: found job 16966519 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9d6aabb509d9742e27b02f1ef4b7f9d6 with result from task 142
parsl.dataflow.dflow DEBUG: Task 142 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/5e69a2d3-0f93-4f86-a3f5-b2b62b6cb70a_isrForCrosstalkSources_855716_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/5e69a2d3-0f93-4f86-a3f5-b2b62b6cb70a_isrForCrosstalkSources_855716_2.stderr
parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 15899c6a90bed419dfb2a8aab2524e82 with result from task 144
parsl.dataflow.dflow DEBUG: Task 144 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/df2105a6-d915-46d4-a931-606ad282fc8a_isrForCrosstalkSources_855718_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/df2105a6-d915-46d4-a931-606ad282fc8a_isrForCrosstalkSources_855718_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:52 proc_lsst.multi:146 [INFO]  found job 16966520 in provider escience_a40
proc_lsst.multi INFO: found job 16966520 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:52 proc_lsst.multi:146 [INFO]  found job 16966521 in provider escience_a40
proc_lsst.multi INFO: found job 16966521 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:52 proc_lsst.multi:146 [INFO]  found job 16966522 in provider escience_a40
proc_lsst.multi INFO: found job 16966522 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:52 proc_lsst.multi:146 [INFO]  found job 16966523 in provider escience_a40
proc_lsst.multi INFO: found job 16966523 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:52 proc_lsst.multi:146 [INFO]  found job 16966524 in provider escience_a40
proc_lsst.multi INFO: found job 16966524 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 425184cc0449a6e8cf1e22d49b165f3d with result from task 140
parsl.dataflow.dflow DEBUG: Task 140 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/a8499962-f6a5-40c2-a1ae-36336f8ee50e_isrForCrosstalkSources_855809_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/a8499962-f6a5-40c2-a1ae-36336f8ee50e_isrForCrosstalkSources_855809_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:53 proc_lsst.multi:146 [INFO]  found job 16966525 in provider escience_a40
proc_lsst.multi INFO: found job 16966525 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:53 proc_lsst.multi:146 [INFO]  found job 16966526 in provider escience_a40
proc_lsst.multi INFO: found job 16966526 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:53 proc_lsst.multi:146 [INFO]  found job 16966527 in provider escience_a40
proc_lsst.multi INFO: found job 16966527 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:53 proc_lsst.multi:146 [INFO]  found job 16966528 in provider escience_a40
proc_lsst.multi INFO: found job 16966528 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:53 proc_lsst.multi:146 [INFO]  found job 16966529 in provider escience_a40
proc_lsst.multi INFO: found job 16966529 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:54 proc_lsst.multi:146 [INFO]  found job 16966530 in provider escience_a40
proc_lsst.multi INFO: found job 16966530 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:54 proc_lsst.multi:146 [INFO]  found job 16966531 in provider escience_a40
proc_lsst.multi INFO: found job 16966531 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:54 proc_lsst.multi:146 [INFO]  found job 16966532 in provider escience_a40
proc_lsst.multi INFO: found job 16966532 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:54 proc_lsst.multi:146 [INFO]  found job 16966533 in provider escience_a40
proc_lsst.multi INFO: found job 16966533 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:54 proc_lsst.multi:146 [INFO]  found job 16966534 in provider escience_a40
proc_lsst.multi INFO: found job 16966534 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:55 proc_lsst.multi:146 [INFO]  found job 16966535 in provider checkpoint
proc_lsst.multi INFO: found job 16966535 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:55 proc_lsst.multi:146 [INFO]  found job 16966536 in provider checkpoint
proc_lsst.multi INFO: found job 16966536 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:55 proc_lsst.multi:146 [INFO]  found job 16966537 in provider checkpoint
proc_lsst.multi INFO: found job 16966537 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:55 proc_lsst.multi:146 [INFO]  found job 16966538 in provider checkpoint
proc_lsst.multi INFO: found job 16966538 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:55 proc_lsst.multi:146 [INFO]  found job 16966539 in provider checkpoint
proc_lsst.multi INFO: found job 16966539 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:56 proc_lsst.multi:146 [INFO]  found job 16966540 in provider checkpoint
proc_lsst.multi INFO: found job 16966540 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:56 proc_lsst.multi:146 [INFO]  found job 16966541 in provider checkpoint
proc_lsst.multi INFO: found job 16966541 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:56 proc_lsst.multi:146 [INFO]  found job 16966542 in provider checkpoint
proc_lsst.multi INFO: found job 16966542 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:56 proc_lsst.multi:146 [INFO]  found job 16966543 in provider checkpoint
proc_lsst.multi INFO: found job 16966543 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:57 proc_lsst.multi:146 [INFO]  found job 16966544 in provider checkpoint
proc_lsst.multi INFO: found job 16966544 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:57 proc_lsst.multi:146 [INFO]  found job 16966545 in provider checkpoint
proc_lsst.multi INFO: found job 16966545 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:57 proc_lsst.multi:146 [INFO]  found job 16966546 in provider checkpoint
proc_lsst.multi INFO: found job 16966546 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:57 proc_lsst.multi:146 [INFO]  found job 16966547 in provider checkpoint
proc_lsst.multi INFO: found job 16966547 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:57 proc_lsst.multi:146 [INFO]  found job 16966548 in provider checkpoint
proc_lsst.multi INFO: found job 16966548 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:58 proc_lsst.multi:146 [INFO]  found job 16966549 in provider checkpoint
proc_lsst.multi INFO: found job 16966549 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:58 proc_lsst.multi:146 [INFO]  found job 16966550 in provider checkpoint
proc_lsst.multi INFO: found job 16966550 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:58 proc_lsst.multi:146 [INFO]  found job 16966551 in provider checkpoint
proc_lsst.multi INFO: found job 16966551 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:58 proc_lsst.multi:146 [INFO]  found job 16966553 in provider checkpoint
proc_lsst.multi INFO: found job 16966553 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:58 proc_lsst.multi:146 [INFO]  found job 16966554 in provider checkpoint
proc_lsst.multi INFO: found job 16966554 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:59 proc_lsst.multi:146 [INFO]  found job 16966555 in provider checkpoint
proc_lsst.multi INFO: found job 16966555 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:59 proc_lsst.multi:146 [INFO]  found job 16966556 in provider checkpoint
proc_lsst.multi INFO: found job 16966556 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:59 proc_lsst.multi:146 [INFO]  found job 16966557 in provider checkpoint
proc_lsst.multi INFO: found job 16966557 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:59 proc_lsst.multi:146 [INFO]  found job 16966558 in provider checkpoint
proc_lsst.multi INFO: found job 16966558 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:37:59 proc_lsst.multi:146 [INFO]  found job 16966559 in provider checkpoint
proc_lsst.multi INFO: found job 16966559 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:00 proc_lsst.multi:146 [INFO]  found job 16966560 in provider checkpoint
proc_lsst.multi INFO: found job 16966560 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 10e018a18c26db9b0ba4a408c452ffdc with result from task 173
parsl.dataflow.dflow DEBUG: Task 173 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/78be7c6b-f527-4f2c-9b5f-4feba01f7a66_isrForCrosstalkSources_855810_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/78be7c6b-f527-4f2c-9b5f-4feba01f7a66_isrForCrosstalkSources_855810_12.stderr
parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 240fe90bc0be4c87f1a786356f243d8b with result from task 169
parsl.dataflow.dflow DEBUG: Task 169 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/3e6ea057-1857-4ad0-9b66-5d391c48954c_isrForCrosstalkSources_855717_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/3e6ea057-1857-4ad0-9b66-5d391c48954c_isrForCrosstalkSources_855717_51.stderr
parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4f534cd45095148e4781b62cae858e76 with result from task 172
parsl.dataflow.dflow DEBUG: Task 172 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/abb76991-ffc2-46e5-a630-8c913de49c8d_isrForCrosstalkSources_855809_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/abb76991-ffc2-46e5-a630-8c913de49c8d_isrForCrosstalkSources_855809_7.stderr
parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f4adfc1013f0f12252939d348b70c2f8 with result from task 166
parsl.dataflow.dflow DEBUG: Task 166 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/350e2e1e-bf22-44e7-a3e7-6011af902c86_isrForCrosstalkSources_855809_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/350e2e1e-bf22-44e7-a3e7-6011af902c86_isrForCrosstalkSources_855809_18.stderr
parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 921b30cac6b55b5b1f1874835c2ce8e1 with result from task 168
parsl.dataflow.dflow DEBUG: Task 168 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/799d0c18-366e-4a95-a26d-3a9bc49aaf10_isrForCrosstalkSources_855809_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/799d0c18-366e-4a95-a26d-3a9bc49aaf10_isrForCrosstalkSources_855809_17.stderr
parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2e11e46fb9469c3c2019ed67263f204c with result from task 162
parsl.dataflow.dflow DEBUG: Task 162 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/48569b30-1dc4-4a76-9b28-9d64457f17fc_isrForCrosstalkSources_855716_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/48569b30-1dc4-4a76-9b28-9d64457f17fc_isrForCrosstalkSources_855716_27.stderr
parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 316e89bfc1be86236fc59e3339f1b558 with result from task 167
parsl.dataflow.dflow DEBUG: Task 167 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/7fb19a06-7f24-4afb-9be0-ba015f845bd7_isrForCrosstalkSources_855810_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/7fb19a06-7f24-4afb-9be0-ba015f845bd7_isrForCrosstalkSources_855810_1.stderr
parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ec6b1aeccaa90c57f2088ed665ec2530 with result from task 171
parsl.dataflow.dflow DEBUG: Task 171 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/99b10e52-d06b-44a2-b4b6-1157486b9ea7_isrForCrosstalkSources_855716_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/99b10e52-d06b-44a2-b4b6-1157486b9ea7_isrForCrosstalkSources_855716_34.stderr
parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 981fc4919ee5b26dce83655b02b6a366 with result from task 175
parsl.dataflow.dflow DEBUG: Task 175 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/6080d48e-23e8-4405-964c-3507e295bba9_isrForCrosstalkSources_855809_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/6080d48e-23e8-4405-964c-3507e295bba9_isrForCrosstalkSources_855809_32.stderr
parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 78814ef32036b48919551dd40bcac938 with result from task 174
parsl.dataflow.dflow DEBUG: Task 174 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/0646698b-954e-49f0-9fda-8962d99cec65_isrForCrosstalkSources_855808_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/0646698b-954e-49f0-9fda-8962d99cec65_isrForCrosstalkSources_855808_36.stderr
parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ea584ee5c48fc119ccbc9f27d5b75dd1 with result from task 164
parsl.dataflow.dflow DEBUG: Task 164 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/e7f20a5c-b75f-4512-8b8c-4008ecccdaf7_isrForCrosstalkSources_855810_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/e7f20a5c-b75f-4512-8b8c-4008ecccdaf7_isrForCrosstalkSources_855810_57.stderr
parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 17a510e84afd1b9e7113bc84a4605ba0 with result from task 165
parsl.dataflow.dflow DEBUG: Task 165 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/5d81e525-ad85-42c7-9608-4f46af9a757c_isrForCrosstalkSources_855810_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/5d81e525-ad85-42c7-9608-4f46af9a757c_isrForCrosstalkSources_855810_18.stderr
parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cef28d19823f7bce6d5477cd0d03fc5f with result from task 161
parsl.dataflow.dflow DEBUG: Task 161 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/59a28fba-e9c7-4873-a397-145f46f3c80c_isrForCrosstalkSources_855716_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/59a28fba-e9c7-4873-a397-145f46f3c80c_isrForCrosstalkSources_855716_37.stderr
parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5bec02355d4f0f0d21fd4fe5fa3e9ac8 with result from task 176
parsl.dataflow.dflow DEBUG: Task 176 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/46fb84b8-b959-4285-a793-a85298667fb2_isrForCrosstalkSources_855808_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/46fb84b8-b959-4285-a793-a85298667fb2_isrForCrosstalkSources_855808_42.stderr
parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cf2dee9a2d2688af0400eb4022a1eab8 with result from task 170
parsl.dataflow.dflow DEBUG: Task 170 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/ce5fdca5-5516-4c06-9cde-a4f94a16a37a_isrForCrosstalkSources_855808_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/ce5fdca5-5516-4c06-9cde-a4f94a16a37a_isrForCrosstalkSources_855808_52.stderr
parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a06a6a364a46330e4ddf15d3a69c7c46 with result from task 163
parsl.dataflow.dflow DEBUG: Task 163 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/122350dc-a15f-4b57-9da0-70f97ff95102_isrForCrosstalkSources_855716_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/122350dc-a15f-4b57-9da0-70f97ff95102_isrForCrosstalkSources_855716_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:00 proc_lsst.multi:146 [INFO]  found job 16966561 in provider checkpoint
proc_lsst.multi INFO: found job 16966561 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:00 proc_lsst.multi:146 [INFO]  found job 16966562 in provider checkpoint
proc_lsst.multi INFO: found job 16966562 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:00 proc_lsst.multi:146 [INFO]  found job 16966563 in provider checkpoint
proc_lsst.multi INFO: found job 16966563 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:00 proc_lsst.multi:146 [INFO]  found job 16966564 in provider checkpoint
proc_lsst.multi INFO: found job 16966564 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:01 proc_lsst.multi:146 [INFO]  found job 16966565 in provider checkpoint
proc_lsst.multi INFO: found job 16966565 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:01 proc_lsst.multi:146 [INFO]  found job 16966566 in provider checkpoint
proc_lsst.multi INFO: found job 16966566 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:01 proc_lsst.multi:146 [INFO]  found job 16966567 in provider checkpoint
proc_lsst.multi INFO: found job 16966567 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:01 proc_lsst.multi:146 [INFO]  found job 16966568 in provider checkpoint
proc_lsst.multi INFO: found job 16966568 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:01 proc_lsst.multi:146 [INFO]  found job 16966569 in provider checkpoint
proc_lsst.multi INFO: found job 16966569 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:02 proc_lsst.multi:146 [INFO]  found job 16966570 in provider checkpoint
proc_lsst.multi INFO: found job 16966570 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:02 proc_lsst.multi:146 [INFO]  found job 16966571 in provider checkpoint
proc_lsst.multi INFO: found job 16966571 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:02 proc_lsst.multi:146 [INFO]  found job 16966572 in provider checkpoint
proc_lsst.multi INFO: found job 16966572 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 145
parsl.dataflow.strategy DEBUG: Executor multi has 145 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23383474330768
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 145
parsl.dataflow.strategy DEBUG: Executor multi has 145 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23383474330768
parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2d6ea3cb8a3f8d1dacce730841ebc208 with result from task 253
parsl.dataflow.dflow DEBUG: Task 253 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/97585b2f-c575-4912-b0e0-adf3ce82f49f_isrForCrosstalkSources_855718_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/97585b2f-c575-4912-b0e0-adf3ce82f49f_isrForCrosstalkSources_855718_25.stderr
parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 91568c03361363d852dfd50672382936 with result from task 257
parsl.dataflow.dflow DEBUG: Task 257 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/2e6195ed-c15a-4ee2-861a-d26e040fee73_isrForCrosstalkSources_855808_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/2e6195ed-c15a-4ee2-861a-d26e040fee73_isrForCrosstalkSources_855808_55.stderr
parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d27181e60f36f0d2e838762d3ae515b0 with result from task 252
parsl.dataflow.dflow DEBUG: Task 252 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/1d17a912-cbb7-4bd9-b336-ddbb556e1e7d_isrForCrosstalkSources_855809_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/1d17a912-cbb7-4bd9-b336-ddbb556e1e7d_isrForCrosstalkSources_855809_27.stderr
parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 34f2ac42e99e0a854c4d45a9a248fd5b with result from task 258
parsl.dataflow.dflow DEBUG: Task 258 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/fc16ea36-d4fe-4ad3-badf-06bbb8ba1eba_isrForCrosstalkSources_855809_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/fc16ea36-d4fe-4ad3-badf-06bbb8ba1eba_isrForCrosstalkSources_855809_8.stderr
parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ecd3d9ffe4a0ef41ee3d1b7786500ab8 with result from task 254
parsl.dataflow.dflow DEBUG: Task 254 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/bb478b12-7bb5-4b0b-8463-8761244c2cea_isrForCrosstalkSources_855716_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/bb478b12-7bb5-4b0b-8463-8761244c2cea_isrForCrosstalkSources_855716_10.stderr
parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d4508bb2bf580a702aaa5221740d1b4e with result from task 255
parsl.dataflow.dflow DEBUG: Task 255 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/60321cae-52e8-405b-8f2e-0442f93caa80_isrForCrosstalkSources_855809_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/60321cae-52e8-405b-8f2e-0442f93caa80_isrForCrosstalkSources_855809_36.stderr
parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 845de65f493c8d3fda3adc9e97a4c798 with result from task 256
parsl.dataflow.dflow DEBUG: Task 256 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/a4f86ebe-87c4-4b2e-9aca-f5ff146cbbee_isrForCrosstalkSources_855716_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/a4f86ebe-87c4-4b2e-9aca-f5ff146cbbee_isrForCrosstalkSources_855716_61.stderr
parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 59619b3a5044affe6c7ffa32948c2e55 with result from task 259
parsl.dataflow.dflow DEBUG: Task 259 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/d2534519-81b6-462f-b5b4-4d7388e26bf9_isrForCrosstalkSources_855810_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/d2534519-81b6-462f-b5b4-4d7388e26bf9_isrForCrosstalkSources_855810_6.stderr
parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 182e02df5cce2846203c1dcac822bd15 with result from task 208
parsl.dataflow.dflow DEBUG: Task 208 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/c969b9c3-6083-486a-ab80-81e8c09bb908_isrForCrosstalkSources_855808_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/c969b9c3-6083-486a-ab80-81e8c09bb908_isrForCrosstalkSources_855808_28.stderr
parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6652b7e29e94530345e7db7ce0820a4e with result from task 203
parsl.dataflow.dflow DEBUG: Task 203 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/984f5f95-1514-4f98-bd51-c4d39048f787_isrForCrosstalkSources_855717_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/984f5f95-1514-4f98-bd51-c4d39048f787_isrForCrosstalkSources_855717_17.stderr
parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eb039b3137cc46c88b56e111a4732ca6 with result from task 207
parsl.dataflow.dflow DEBUG: Task 207 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/99fa8782-0c6c-4ac4-9a22-078474c96644_isrForCrosstalkSources_855810_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/99fa8782-0c6c-4ac4-9a22-078474c96644_isrForCrosstalkSources_855810_4.stderr
parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f9822fffdcebe75ad49a8da6c24e7e4c with result from task 200
parsl.dataflow.dflow DEBUG: Task 200 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/bb75a2d7-b156-4b60-a12d-2f167e50792b_isrForCrosstalkSources_855808_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/bb75a2d7-b156-4b60-a12d-2f167e50792b_isrForCrosstalkSources_855808_34.stderr
parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 984698830d50fd3fb0b0c1308520115a with result from task 201
parsl.dataflow.dflow DEBUG: Task 201 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/c9e0836b-0b17-4787-84ed-c0e1332b0ff3_isrForCrosstalkSources_855809_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/c9e0836b-0b17-4787-84ed-c0e1332b0ff3_isrForCrosstalkSources_855809_25.stderr
parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 12757c7e988448e1912eb07ef1e661d4 with result from task 195
parsl.dataflow.dflow DEBUG: Task 195 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/a145f8e5-73ad-4559-9599-725daec5614a_isrForCrosstalkSources_855809_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/a145f8e5-73ad-4559-9599-725daec5614a_isrForCrosstalkSources_855809_35.stderr
parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 83698a131a77853123eb319e86c37258 with result from task 198
parsl.dataflow.dflow DEBUG: Task 198 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/9881421b-55b5-469e-8422-fe0f7ab78f2d_isrForCrosstalkSources_855808_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/9881421b-55b5-469e-8422-fe0f7ab78f2d_isrForCrosstalkSources_855808_37.stderr
parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ca6a6f668281c7147cccf5d29035a657 with result from task 209
parsl.dataflow.dflow DEBUG: Task 209 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/9ac6ffd9-99c3-4dc2-a98c-4de28a43e654_isrForCrosstalkSources_855717_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/9ac6ffd9-99c3-4dc2-a98c-4de28a43e654_isrForCrosstalkSources_855717_2.stderr
parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7f3ac8a44ec72b3696f7e1c52d7ca6b6 with result from task 210
parsl.dataflow.dflow DEBUG: Task 210 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/a855c5c2-ec39-4a78-9618-7d610818d0a7_isrForCrosstalkSources_855808_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/a855c5c2-ec39-4a78-9618-7d610818d0a7_isrForCrosstalkSources_855808_51.stderr
parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c99f35b7aeb6612b8677bf3de600eee3 with result from task 206
parsl.dataflow.dflow DEBUG: Task 206 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/a1247340-9c34-4a2e-bbf3-c06eb68b02e5_isrForCrosstalkSources_855717_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/a1247340-9c34-4a2e-bbf3-c06eb68b02e5_isrForCrosstalkSources_855717_59.stderr
parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 72af6055041e37385ca6a9150e5af222 with result from task 205
parsl.dataflow.dflow DEBUG: Task 205 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/52ae9c61-1023-4d17-b911-11ce77ba23b0_isrForCrosstalkSources_855810_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/52ae9c61-1023-4d17-b911-11ce77ba23b0_isrForCrosstalkSources_855810_44.stderr
parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f5fa84e01fd499469595d51b20b3bc01 with result from task 197
parsl.dataflow.dflow DEBUG: Task 197 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/9e758c56-2eb6-45be-b405-de6c26920628_isrForCrosstalkSources_855810_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/9e758c56-2eb6-45be-b405-de6c26920628_isrForCrosstalkSources_855810_30.stderr
parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4cef4df281c047b87cedaf0229d61082 with result from task 204
parsl.dataflow.dflow DEBUG: Task 204 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/3369a009-484e-419e-b499-76dbffc4bf77_isrForCrosstalkSources_855717_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/3369a009-484e-419e-b499-76dbffc4bf77_isrForCrosstalkSources_855717_43.stderr
parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c334031620255acd9eb74e1b283a8d24 with result from task 194
parsl.dataflow.dflow DEBUG: Task 194 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/1857a2e6-37c5-45ae-ac61-ac17eff4aebd_isrForCrosstalkSources_855718_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/1857a2e6-37c5-45ae-ac61-ac17eff4aebd_isrForCrosstalkSources_855718_6.stderr
parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a89abac441908b032eb8e932c2f433d6 with result from task 196
parsl.dataflow.dflow DEBUG: Task 196 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/e0882b93-5cd9-4dfd-ba5f-a4fedcbc2c1c_isrForCrosstalkSources_855718_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/e0882b93-5cd9-4dfd-ba5f-a4fedcbc2c1c_isrForCrosstalkSources_855718_49.stderr
parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry edb021ce91ccffb510e0c0b4f977aaa3 with result from task 199
parsl.dataflow.dflow DEBUG: Task 199 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/16b1ec3d-4433-41e8-b231-86e6b48aed09_isrForCrosstalkSources_855717_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/16b1ec3d-4433-41e8-b231-86e6b48aed09_isrForCrosstalkSources_855717_40.stderr
parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2b33e545bb1ba69f9ca7387c4f79d43f with result from task 217
parsl.dataflow.dflow DEBUG: Task 217 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/24daf74d-17fb-426f-b223-4fcc94ace12f_isrForCrosstalkSources_855716_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/24daf74d-17fb-426f-b223-4fcc94ace12f_isrForCrosstalkSources_855716_9.stderr
parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4cfc427cc7d8426caf7bad7c0ea13362 with result from task 219
parsl.dataflow.dflow DEBUG: Task 219 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/3d4c6612-d6b1-45bf-af3d-139be4f82eb4_isrForCrosstalkSources_855808_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/3d4c6612-d6b1-45bf-af3d-139be4f82eb4_isrForCrosstalkSources_855808_50.stderr
parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 02b8749ec7d60afb561d50ad22c77f6b with result from task 220
parsl.dataflow.dflow DEBUG: Task 220 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/7fbf74d1-8679-4d5d-a5c9-0416c7e689ba_isrForCrosstalkSources_855809_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/7fbf74d1-8679-4d5d-a5c9-0416c7e689ba_isrForCrosstalkSources_855809_62.stderr
parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6caefd252b34cb9b4a7f33148d5992c0 with result from task 218
parsl.dataflow.dflow DEBUG: Task 218 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/401dfa2d-9d29-4e35-86a6-ca3d24b99b03_isrForCrosstalkSources_855718_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/401dfa2d-9d29-4e35-86a6-ca3d24b99b03_isrForCrosstalkSources_855718_33.stderr
parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7ebcf9f5818bbe5733383c0eeda8f477 with result from task 216
parsl.dataflow.dflow DEBUG: Task 216 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/1a1f3922-8fa1-45ff-a6b7-ce85175a8cb0_isrForCrosstalkSources_855717_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/1a1f3922-8fa1-45ff-a6b7-ce85175a8cb0_isrForCrosstalkSources_855717_34.stderr
parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fd2336b5816e7f56c3261b6c1cea2c20 with result from task 215
parsl.dataflow.dflow DEBUG: Task 215 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/edd140f5-cb14-44d1-aad4-6494c6830de3_isrForCrosstalkSources_855809_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/edd140f5-cb14-44d1-aad4-6494c6830de3_isrForCrosstalkSources_855809_21.stderr
parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 78745bb48b76168ba1885f3be067f698 with result from task 214
parsl.dataflow.dflow DEBUG: Task 214 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/9ace02a0-85cf-4596-8726-61d2c5cc37c6_isrForCrosstalkSources_855808_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/9ace02a0-85cf-4596-8726-61d2c5cc37c6_isrForCrosstalkSources_855808_16.stderr
parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f7822698e80c0a0fe1ea928233253acb with result from task 221
parsl.dataflow.dflow DEBUG: Task 221 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/888f8c68-5adf-471b-b46a-aaf791f8e3c9_isrForCrosstalkSources_855717_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/888f8c68-5adf-471b-b46a-aaf791f8e3c9_isrForCrosstalkSources_855717_60.stderr
parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 77b9089161cf2c47d669a4a75bfa91a5 with result from task 246
parsl.dataflow.dflow DEBUG: Task 246 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/5534f9e9-13fc-48b7-9280-ebd73fa59f67_isrForCrosstalkSources_855809_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/5534f9e9-13fc-48b7-9280-ebd73fa59f67_isrForCrosstalkSources_855809_29.stderr
parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 21ef9ace09101e3492217799792947b8 with result from task 249
parsl.dataflow.dflow DEBUG: Task 249 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/7e39db8d-6c21-4648-81cc-9298a6e73ae2_isrForCrosstalkSources_855716_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/7e39db8d-6c21-4648-81cc-9298a6e73ae2_isrForCrosstalkSources_855716_12.stderr
parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0d51a2e23b48ce6dee27848e50944b11 with result from task 247
parsl.dataflow.dflow DEBUG: Task 247 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/a453a728-fc1e-4391-9a9e-07c287b12884_isrForCrosstalkSources_855809_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/a453a728-fc1e-4391-9a9e-07c287b12884_isrForCrosstalkSources_855809_53.stderr
parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4efd9164115527fca491e26dadf0f542 with result from task 248
parsl.dataflow.dflow DEBUG: Task 248 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/c31ea1dc-76e9-454b-92d7-50e9b0a58ab0_isrForCrosstalkSources_855718_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/c31ea1dc-76e9-454b-92d7-50e9b0a58ab0_isrForCrosstalkSources_855718_15.stderr
parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9a6866419970beb4ced0c5d70f9f093e with result from task 243
parsl.dataflow.dflow DEBUG: Task 243 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/2a7bc524-c03f-481f-9115-b3aab782bc25_isrForCrosstalkSources_855717_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/2a7bc524-c03f-481f-9115-b3aab782bc25_isrForCrosstalkSources_855717_16.stderr
parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 263ccaeaf07b3f70ef48df6a6aab9f1c with result from task 244
parsl.dataflow.dflow DEBUG: Task 244 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/7059036c-46a2-46d7-907c-d71ee72eae55_isrForCrosstalkSources_855718_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/7059036c-46a2-46d7-907c-d71ee72eae55_isrForCrosstalkSources_855718_11.stderr
parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e8eeb468565663d7562d0eb7c272b766 with result from task 242
parsl.dataflow.dflow DEBUG: Task 242 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/40e8b8cd-3d33-4d0d-b7fc-11a1f59b3cae_isrForCrosstalkSources_855717_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/40e8b8cd-3d33-4d0d-b7fc-11a1f59b3cae_isrForCrosstalkSources_855717_31.stderr
parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry db033121155852952332113a98f41491 with result from task 245
parsl.dataflow.dflow DEBUG: Task 245 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/168911cb-16fc-43c1-ad8b-2a19170ea71f_isrForCrosstalkSources_855810_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/168911cb-16fc-43c1-ad8b-2a19170ea71f_isrForCrosstalkSources_855810_20.stderr
parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 10cc73149eb125d2ef74a977e19ece69 with result from task 277
parsl.dataflow.dflow DEBUG: Task 277 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/f1ebf6ab-3ee6-4d48-8211-f5c4b4bb6b8f_isrForCrosstalkSources_855810_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/f1ebf6ab-3ee6-4d48-8211-f5c4b4bb6b8f_isrForCrosstalkSources_855810_52.stderr
parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5efeb48e424f6bb67f46ce38816fc6ca with result from task 281
parsl.dataflow.dflow DEBUG: Task 281 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/9c110c98-5a5e-4eb2-864b-03026bf6f05f_isrForCrosstalkSources_855808_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/9c110c98-5a5e-4eb2-864b-03026bf6f05f_isrForCrosstalkSources_855808_43.stderr
parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9271690015a42d53ae04b041a4fa3812 with result from task 275
parsl.dataflow.dflow DEBUG: Task 275 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/c749752a-c3ab-4631-92ed-b86cc0635446_isrForCrosstalkSources_855809_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/c749752a-c3ab-4631-92ed-b86cc0635446_isrForCrosstalkSources_855809_19.stderr
parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d9e70aebb933a8751997c06c6c24d460 with result from task 279
parsl.dataflow.dflow DEBUG: Task 279 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/b18c6f0b-c71c-4ba5-a257-fca9251ab8ba_isrForCrosstalkSources_855716_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/b18c6f0b-c71c-4ba5-a257-fca9251ab8ba_isrForCrosstalkSources_855716_31.stderr
parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5302c60787d64d54e572e082aed3ce27 with result from task 274
parsl.dataflow.dflow DEBUG: Task 274 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/b465fb30-fe7e-4490-9d9d-0f9f4a51414c_isrForCrosstalkSources_855809_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/b465fb30-fe7e-4490-9d9d-0f9f4a51414c_isrForCrosstalkSources_855809_58.stderr
parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cb43c32838e2452a55d0ac1e258d59c3 with result from task 276
parsl.dataflow.dflow DEBUG: Task 276 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/60ffea92-d07b-4a7f-a41c-a509c2fd31f1_isrForCrosstalkSources_855808_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/60ffea92-d07b-4a7f-a41c-a509c2fd31f1_isrForCrosstalkSources_855808_53.stderr
parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bac429c4b44718b159b96f653d222365 with result from task 278
parsl.dataflow.dflow DEBUG: Task 278 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/066fc771-8b2f-423a-9040-080a7b358439_isrForCrosstalkSources_855809_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/066fc771-8b2f-423a-9040-080a7b358439_isrForCrosstalkSources_855809_5.stderr
parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b8134fb82c3f187795e89e5cc225a0ce with result from task 280
parsl.dataflow.dflow DEBUG: Task 280 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/c6631729-423b-4c67-824d-0d8fb757aaaa_isrForCrosstalkSources_855717_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/c6631729-423b-4c67-824d-0d8fb757aaaa_isrForCrosstalkSources_855717_12.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 97
parsl.dataflow.strategy DEBUG: Executor multi has 97 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23383474330768
parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d215b1cb9081473e603cb862f9ae79b7 with result from task 229
parsl.dataflow.dflow DEBUG: Task 229 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/9b108f64-3ff0-40e3-a2d7-56a4a54d4d9f_isrForCrosstalkSources_855808_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/9b108f64-3ff0-40e3-a2d7-56a4a54d4d9f_isrForCrosstalkSources_855808_24.stderr
parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 56f2beae56dd12a8cbebe7d60d779aaa with result from task 231
parsl.dataflow.dflow DEBUG: Task 231 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/da3efb51-9f28-4749-88f3-49fcda3e126c_isrForCrosstalkSources_855810_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/da3efb51-9f28-4749-88f3-49fcda3e126c_isrForCrosstalkSources_855810_14.stderr
parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2b77af1578ad1a650015aa90b34aef6d with result from task 228
parsl.dataflow.dflow DEBUG: Task 228 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/8ba77c77-1557-414d-9cd3-67707fcd7a51_isrForCrosstalkSources_855718_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/8ba77c77-1557-414d-9cd3-67707fcd7a51_isrForCrosstalkSources_855718_23.stderr
parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8b83c68b44a0aaf2bbb92cdceb1e9045 with result from task 233
parsl.dataflow.dflow DEBUG: Task 233 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/4389265c-f505-4f6b-bb57-f9a5da7c598a_isrForCrosstalkSources_855809_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/4389265c-f505-4f6b-bb57-f9a5da7c598a_isrForCrosstalkSources_855809_59.stderr
parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3159b3273378281d5f382e5b7b3d3394 with result from task 232
parsl.dataflow.dflow DEBUG: Task 232 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/3688284c-4bec-4b9e-a7af-7bf3a93917f0_isrForCrosstalkSources_855809_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/3688284c-4bec-4b9e-a7af-7bf3a93917f0_isrForCrosstalkSources_855809_30.stderr
parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 50a0273c82e6c26c0fcff82f50891ae8 with result from task 226
parsl.dataflow.dflow DEBUG: Task 226 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/a4a37ddc-61f5-42af-8c98-c99dfaaf6f32_isrForCrosstalkSources_855717_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/a4a37ddc-61f5-42af-8c98-c99dfaaf6f32_isrForCrosstalkSources_855717_47.stderr
parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 12328fa5f160127db85e8f191bb5f6e8 with result from task 227
parsl.dataflow.dflow DEBUG: Task 227 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/005d275e-e745-4145-957b-91a9cfa63aaf_isrForCrosstalkSources_855717_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/005d275e-e745-4145-957b-91a9cfa63aaf_isrForCrosstalkSources_855717_14.stderr
parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d7ec21c67296ca49cf9cf5e9929b9581 with result from task 230
parsl.dataflow.dflow DEBUG: Task 230 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/a971e37d-e898-438b-832c-b30cb2c10066_isrForCrosstalkSources_855716_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/a971e37d-e898-438b-832c-b30cb2c10066_isrForCrosstalkSources_855716_28.stderr
parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9eb9c72536284b51eebf769032809bd2 with result from task 303
parsl.dataflow.dflow DEBUG: Task 303 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/5213aaa5-fdc0-4bff-baa7-c799501c3e9a_isrForCrosstalkSources_855809_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/5213aaa5-fdc0-4bff-baa7-c799501c3e9a_isrForCrosstalkSources_855809_51.stderr
parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 38287291ee99bc0a4e06f5088129bcab with result from task 302
parsl.dataflow.dflow DEBUG: Task 302 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/4dc23d5b-78b4-476b-9d05-a35a436d7b2d_isrForCrosstalkSources_855716_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/4dc23d5b-78b4-476b-9d05-a35a436d7b2d_isrForCrosstalkSources_855716_60.stderr
parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f2a31cb5d1dd780703a31be6beacf52d with result from task 299
parsl.dataflow.dflow DEBUG: Task 299 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/4901c0ad-80ac-407a-8816-c58c4f501aa7_isrForCrosstalkSources_855716_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/4901c0ad-80ac-407a-8816-c58c4f501aa7_isrForCrosstalkSources_855716_47.stderr
parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ca8f64c67dfae9fccfef6d939a00cb5e with result from task 304
parsl.dataflow.dflow DEBUG: Task 304 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/c285269f-0b30-4aa7-aec8-470da365913f_isrForCrosstalkSources_855716_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/c285269f-0b30-4aa7-aec8-470da365913f_isrForCrosstalkSources_855716_15.stderr
2024-03-09 17:38:12 proc_lsst.multi:146 [INFO]  found job 64340 in provider local
proc_lsst.multi INFO: found job 64340 in provider local
2024-03-09 17:38:12 proc_lsst.multi:146 [INFO]  found job 16966475 in provider astro
proc_lsst.multi INFO: found job 16966475 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 19cc75c7bd87f23d0697ebae4a29908d with result from task 301
parsl.dataflow.dflow DEBUG: Task 301 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/bf6612c4-94b8-4d1e-b382-d921dd09a56e_isrForCrosstalkSources_855808_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/bf6612c4-94b8-4d1e-b382-d921dd09a56e_isrForCrosstalkSources_855808_3.stderr
parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ff260bb047ed2981421d1cc434e17fcd with result from task 306
parsl.dataflow.dflow DEBUG: Task 306 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/3d4c4cee-768e-4a78-be5f-5cf320462ae4_isrForCrosstalkSources_855717_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/3d4c4cee-768e-4a78-be5f-5cf320462ae4_isrForCrosstalkSources_855717_7.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:12 proc_lsst.multi:146 [INFO]  found job 16966476 in provider astro
proc_lsst.multi INFO: found job 16966476 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e889df5d4450569ac41eb3e8df98710d with result from task 305
parsl.dataflow.dflow DEBUG: Task 305 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/61f39293-5225-44ca-8cd1-5790c1017042_isrForCrosstalkSources_855717_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/61f39293-5225-44ca-8cd1-5790c1017042_isrForCrosstalkSources_855717_29.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:13 proc_lsst.multi:146 [INFO]  found job 16966477 in provider astro
proc_lsst.multi INFO: found job 16966477 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:13 proc_lsst.multi:146 [INFO]  found job 16966478 in provider astro
proc_lsst.multi INFO: found job 16966478 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8a84a7f7f9a7192634bee25e20a5278e with result from task 300
parsl.dataflow.dflow DEBUG: Task 300 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/5e638a43-2528-4e0f-853b-1d508737cdfb_isrForCrosstalkSources_855810_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/5e638a43-2528-4e0f-853b-1d508737cdfb_isrForCrosstalkSources_855810_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:13 proc_lsst.multi:146 [INFO]  found job 16966479 in provider astro
proc_lsst.multi INFO: found job 16966479 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:13 proc_lsst.multi:146 [INFO]  found job 16966480 in provider astro
proc_lsst.multi INFO: found job 16966480 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:13 proc_lsst.multi:146 [INFO]  found job 16966481 in provider astro
proc_lsst.multi INFO: found job 16966481 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:14 proc_lsst.multi:146 [INFO]  found job 16966482 in provider astro
proc_lsst.multi INFO: found job 16966482 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:14 proc_lsst.multi:146 [INFO]  found job 16966483 in provider astro
proc_lsst.multi INFO: found job 16966483 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:14 proc_lsst.multi:146 [INFO]  found job 16966484 in provider astro
proc_lsst.multi INFO: found job 16966484 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:14 proc_lsst.multi:146 [INFO]  found job 16966485 in provider astro
proc_lsst.multi INFO: found job 16966485 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5681d94461fe94aa2f1842173095817b with result from task 284
parsl.dataflow.dflow DEBUG: Task 284 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/8386d33a-5f36-4771-a593-76b7c5395275_isrForCrosstalkSources_855717_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/8386d33a-5f36-4771-a593-76b7c5395275_isrForCrosstalkSources_855717_35.stderr
parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cfd6f4cce09acc9e0300de8e4102cd6d with result from task 288
parsl.dataflow.dflow DEBUG: Task 288 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/d773fed9-d4aa-4caa-955c-e477eb17b0b0_isrForCrosstalkSources_855716_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/d773fed9-d4aa-4caa-955c-e477eb17b0b0_isrForCrosstalkSources_855716_20.stderr
parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5c7443de8d1077a3ab46db6a88015645 with result from task 286
parsl.dataflow.dflow DEBUG: Task 286 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/4b001e75-100b-4a1e-aa39-73ad2cccad2a_isrForCrosstalkSources_855810_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/4b001e75-100b-4a1e-aa39-73ad2cccad2a_isrForCrosstalkSources_855810_38.stderr
parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 14345ef658e0c93d784552a7cc243551 with result from task 283
parsl.dataflow.dflow DEBUG: Task 283 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/7be9c78f-9017-4a32-947b-9397ab108c70_isrForCrosstalkSources_855717_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/7be9c78f-9017-4a32-947b-9397ab108c70_isrForCrosstalkSources_855717_30.stderr
parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 46d169fbf2d9ed0cae268dfd360d7e53 with result from task 285
parsl.dataflow.dflow DEBUG: Task 285 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/61c76934-b461-4d8c-8c59-078a59de99b2_isrForCrosstalkSources_855810_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/61c76934-b461-4d8c-8c59-078a59de99b2_isrForCrosstalkSources_855810_32.stderr
parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3f9cc38c61f58417616ea12a48a83076 with result from task 290
parsl.dataflow.dflow DEBUG: Task 290 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/98cf0e9f-c333-4e29-8c19-6347dd316cdd_isrForCrosstalkSources_855718_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/98cf0e9f-c333-4e29-8c19-6347dd316cdd_isrForCrosstalkSources_855718_24.stderr
parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 53951ad58ee7df4b34f1ef41c04d51ef with result from task 289
parsl.dataflow.dflow DEBUG: Task 289 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/868ef362-f9ae-442c-bb94-8a99c7c4c4bd_isrForCrosstalkSources_855717_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/868ef362-f9ae-442c-bb94-8a99c7c4c4bd_isrForCrosstalkSources_855717_5.stderr
parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9fb08a471d1d0586d613bb36fa1bf6d3 with result from task 287
parsl.dataflow.dflow DEBUG: Task 287 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/d6f96736-f9f0-4925-8fb3-97ef2983a4d4_isrForCrosstalkSources_855716_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/d6f96736-f9f0-4925-8fb3-97ef2983a4d4_isrForCrosstalkSources_855716_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:14 proc_lsst.multi:146 [INFO]  found job 16966486 in provider astro
proc_lsst.multi INFO: found job 16966486 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:15 proc_lsst.multi:146 [INFO]  found job 16966487 in provider astro
proc_lsst.multi INFO: found job 16966487 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:15 proc_lsst.multi:146 [INFO]  found job 16966488 in provider astro
proc_lsst.multi INFO: found job 16966488 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 25aaf96f8ec658e604acb25d2b61a8c0 with result from task 310
parsl.dataflow.dflow DEBUG: Task 310 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/f825faf8-41e3-4e86-8c47-70a8ba97aa5c_isrForCrosstalkSources_855810_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/f825faf8-41e3-4e86-8c47-70a8ba97aa5c_isrForCrosstalkSources_855810_23.stderr
parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 36d4c6dd58b89ff0b3a6d8b079ba117b with result from task 311
parsl.dataflow.dflow DEBUG: Task 311 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/80dcbaf1-f4af-48e9-811d-dd105416a76c_isrForCrosstalkSources_855718_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/80dcbaf1-f4af-48e9-811d-dd105416a76c_isrForCrosstalkSources_855718_18.stderr
parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1d705f4a103d7d15713e1c17efa4c624 with result from task 312
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow DEBUG: Task 312 checkpointed
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/77fb20a2-0348-4129-b70b-e43f4fe3556e_isrForCrosstalkSources_855809_56.stdout
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/77fb20a2-0348-4129-b70b-e43f4fe3556e_isrForCrosstalkSources_855809_56.stderr
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:15 proc_lsst.multi:146 [INFO]  found job 16966489 in provider astro
proc_lsst.multi INFO: found job 16966489 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f12e8ab95845731e2cf435e47d096f30 with result from task 314
parsl.dataflow.dflow DEBUG: Task 314 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/3d43dd92-26dd-449a-b077-e3f7934491fb_isrForCrosstalkSources_855808_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/3d43dd92-26dd-449a-b077-e3f7934491fb_isrForCrosstalkSources_855808_40.stderr
parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 84965e5be1c9d29c70824b335db67145 with result from task 308
parsl.dataflow.dflow DEBUG: Task 308 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/8d2eacf2-7d88-45af-b5db-a401e174811c_isrForCrosstalkSources_855716_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/8d2eacf2-7d88-45af-b5db-a401e174811c_isrForCrosstalkSources_855716_21.stderr
parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 94abd6b58c81e3b6a01da4672fe23a0e with result from task 309
parsl.dataflow.dflow DEBUG: Task 309 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/5ceeac75-cf18-469f-9044-5464be0f71bb_isrForCrosstalkSources_855810_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/5ceeac75-cf18-469f-9044-5464be0f71bb_isrForCrosstalkSources_855810_26.stderr
parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4c7b4de85f8855cf204fe3a71bc05803 with result from task 307
parsl.dataflow.dflow DEBUG: Task 307 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/d686ee17-5bcd-4fef-b810-6c77d4d42f06_isrForCrosstalkSources_855810_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/d686ee17-5bcd-4fef-b810-6c77d4d42f06_isrForCrosstalkSources_855810_46.stderr
parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3cd6f4c202f318d769f0781d7cab2348 with result from task 313
parsl.dataflow.dflow DEBUG: Task 313 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/0c576e78-b4e0-4749-a776-7d83e4e4048c_isrForCrosstalkSources_855716_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/0c576e78-b4e0-4749-a776-7d83e4e4048c_isrForCrosstalkSources_855716_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:15 proc_lsst.multi:146 [INFO]  found job 16966490 in provider astro
proc_lsst.multi INFO: found job 16966490 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b0222557dc45768115f9c020e6629526 with result from task 351
parsl.dataflow.dflow DEBUG: Task 351 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/a3842a81-47f2-461a-b430-66ce0523769e_isrForCrosstalkSources_855717_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/a3842a81-47f2-461a-b430-66ce0523769e_isrForCrosstalkSources_855717_24.stderr
parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 436f1c74f0c18dc4f67f8a9a52395a27 with result from task 350
parsl.dataflow.dflow DEBUG: Task 350 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/82d11e33-0cd8-4657-af1e-8f440e8044af_isrForCrosstalkSources_855810_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/82d11e33-0cd8-4657-af1e-8f440e8044af_isrForCrosstalkSources_855810_41.stderr
parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1c438c593b61fd11477e3013df3bb007 with result from task 348
parsl.dataflow.dflow DEBUG: Task 348 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/3e3138ea-ec2f-4b9a-b473-1c8025f86aa1_isrForCrosstalkSources_855810_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/3e3138ea-ec2f-4b9a-b473-1c8025f86aa1_isrForCrosstalkSources_855810_13.stderr
parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8eb4280bde6fb05adeedb7a5a80aedd8 with result from task 352
parsl.dataflow.dflow DEBUG: Task 352 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/4d666a1f-0954-42ce-ac89-95969459291a_isrForCrosstalkSources_855718_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/4d666a1f-0954-42ce-ac89-95969459291a_isrForCrosstalkSources_855718_58.stderr
parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 020153ab774a1be8d792ad2d7112bbd4 with result from task 353
parsl.dataflow.dflow DEBUG: Task 353 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/d6e4f6d8-2497-4df7-abb4-70e6d22aa7b4_isrForCrosstalkSources_855809_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/d6e4f6d8-2497-4df7-abb4-70e6d22aa7b4_isrForCrosstalkSources_855809_14.stderr
parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c82f37dd9a3f1d6f8f45f646de7cbd0d with result from task 347
parsl.dataflow.dflow DEBUG: Task 347 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/af097a2a-d64e-4bc2-95b3-07384d339145_isrForCrosstalkSources_855808_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/af097a2a-d64e-4bc2-95b3-07384d339145_isrForCrosstalkSources_855808_11.stderr
parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 33231b6a3340f764feef49f4afa4fd52 with result from task 346
parsl.dataflow.dflow DEBUG: Task 346 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/b59541fd-6803-4487-9318-5dcaed407b18_isrForCrosstalkSources_855809_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/b59541fd-6803-4487-9318-5dcaed407b18_isrForCrosstalkSources_855809_15.stderr
parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b211f346692d969e719a53b1c3b3826f with result from task 349
parsl.dataflow.dflow DEBUG: Task 349 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/b79c1360-3297-4509-bb61-b8e8d3f42712_isrForCrosstalkSources_855809_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/b79c1360-3297-4509-bb61-b8e8d3f42712_isrForCrosstalkSources_855809_39.stderr
parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ca162c174ddb8c2f23124d30cc4354c0 with result from task 263
parsl.dataflow.dflow DEBUG: Task 263 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/dbae736c-692d-48d8-bdc5-0a5cb32a5eca_isrForCrosstalkSources_855810_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/dbae736c-692d-48d8-bdc5-0a5cb32a5eca_isrForCrosstalkSources_855810_7.stderr
parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 58d09e291c0ea5858284f98ac02aa84c with result from task 265
parsl.dataflow.dflow DEBUG: Task 265 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/6e983234-78dc-48fa-92e3-8d2904318dda_isrForCrosstalkSources_855716_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/6e983234-78dc-48fa-92e3-8d2904318dda_isrForCrosstalkSources_855716_58.stderr
parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ef89a811d8a37aca34c729e3604f125f with result from task 268
parsl.dataflow.dflow DEBUG: Task 268 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/e967644d-bf8d-4223-a761-a521d7c40c89_isrForCrosstalkSources_855716_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/e967644d-bf8d-4223-a761-a521d7c40c89_isrForCrosstalkSources_855716_25.stderr
parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f8ca1035f81685008c1d569a2bb2a6c4 with result from task 262
parsl.dataflow.dflow DEBUG: Task 262 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/4227c6da-cd9c-4898-bf8f-ec1564b250ef_isrForCrosstalkSources_855808_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/4227c6da-cd9c-4898-bf8f-ec1564b250ef_isrForCrosstalkSources_855808_14.stderr
parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 27ed3d6d06696e5656949f784f8a5852 with result from task 264
parsl.dataflow.dflow DEBUG: Task 264 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/4e768d29-bb3f-4553-be41-208a692c1318_isrForCrosstalkSources_855718_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/4e768d29-bb3f-4553-be41-208a692c1318_isrForCrosstalkSources_855718_21.stderr
parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7d4f2c2e40166440372fb59ebcd48b34 with result from task 266
parsl.dataflow.dflow DEBUG: Task 266 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/0becd143-4021-46a4-92ff-6da1703328a6_isrForCrosstalkSources_855718_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/0becd143-4021-46a4-92ff-6da1703328a6_isrForCrosstalkSources_855718_28.stderr
parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 03a8032024819fdbc9b7fef999b560c5 with result from task 267
parsl.dataflow.dflow DEBUG: Task 267 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/1ab74b15-38e0-48bc-a90c-8424730a17f0_isrForCrosstalkSources_855716_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/1ab74b15-38e0-48bc-a90c-8424730a17f0_isrForCrosstalkSources_855716_13.stderr
parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8f73d38401decd94f1ec55d8da6f2127 with result from task 269
parsl.dataflow.dflow DEBUG: Task 269 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/117b2faa-cebf-46cd-9fd3-602ad5235784_isrForCrosstalkSources_855808_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/117b2faa-cebf-46cd-9fd3-602ad5235784_isrForCrosstalkSources_855808_33.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:15 proc_lsst.multi:146 [INFO]  found job 16966491 in provider astro
proc_lsst.multi INFO: found job 16966491 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:16 proc_lsst.multi:146 [INFO]  found job 16966492 in provider astro
proc_lsst.multi INFO: found job 16966492 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:16 proc_lsst.multi:146 [INFO]  found job 16966493 in provider astro
proc_lsst.multi INFO: found job 16966493 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:16 proc_lsst.multi:146 [INFO]  found job 16966499 in provider astro
proc_lsst.multi INFO: found job 16966499 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:16 proc_lsst.multi:146 [INFO]  found job 16966500 in provider astro
proc_lsst.multi INFO: found job 16966500 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:16 proc_lsst.multi:146 [INFO]  found job 16966501 in provider astro
proc_lsst.multi INFO: found job 16966501 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 01b497d10ca9aa461c96aa79c6b5b768 with result from task 317
parsl.dataflow.dflow DEBUG: Task 317 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/d13abbd0-18c5-4efb-b592-32fa1798eee7_isrForCrosstalkSources_855716_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/d13abbd0-18c5-4efb-b592-32fa1798eee7_isrForCrosstalkSources_855716_52.stderr
parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8edcc05786b56a6ba1eb39cd78a54048 with result from task 321
parsl.dataflow.dflow DEBUG: Task 321 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/182cef8e-735d-4230-90c6-191b8b7f7d7a_isrForCrosstalkSources_855809_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/182cef8e-735d-4230-90c6-191b8b7f7d7a_isrForCrosstalkSources_855809_57.stderr
parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ced53466299ed6492c328bf427b89f76 with result from task 320
parsl.dataflow.dflow DEBUG: Task 320 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/359fdaea-3b89-4dcc-9485-01588fa41891_isrForCrosstalkSources_855717_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/359fdaea-3b89-4dcc-9485-01588fa41891_isrForCrosstalkSources_855717_11.stderr
parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1c79b372f8de7683b675fe0892c94153 with result from task 319
parsl.dataflow.dflow DEBUG: Task 319 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/908b3ba5-0565-4ae1-9898-a427de84ffce_isrForCrosstalkSources_855716_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/908b3ba5-0565-4ae1-9898-a427de84ffce_isrForCrosstalkSources_855716_7.stderr
parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 88ae57b4398d8369b1c84f437fb4350f with result from task 316
parsl.dataflow.dflow DEBUG: Task 316 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/0df094f8-7324-4267-89b5-2ce30afa6bb2_isrForCrosstalkSources_855718_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/0df094f8-7324-4267-89b5-2ce30afa6bb2_isrForCrosstalkSources_855718_41.stderr
parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 92133535b28850cdfc3dc58bdc10e025 with result from task 322
parsl.dataflow.dflow DEBUG: Task 322 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/227f0264-3fda-4e3d-b00b-f0fac3a7b510_isrForCrosstalkSources_855716_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/227f0264-3fda-4e3d-b00b-f0fac3a7b510_isrForCrosstalkSources_855716_1.stderr
parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 03eb941eed50e0b00c0877641f3284b1 with result from task 318
parsl.dataflow.dflow DEBUG: Task 318 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/365e7f8e-4159-40d2-aa06-81cc6f0ab058_isrForCrosstalkSources_855810_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/365e7f8e-4159-40d2-aa06-81cc6f0ab058_isrForCrosstalkSources_855810_34.stderr
parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cb7fb2789b57c62b66c3ee43cb657275 with result from task 315
parsl.dataflow.dflow DEBUG: Task 315 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/37928a4d-13b4-4ef5-a590-7a6bd93d072c_isrForCrosstalkSources_855810_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/37928a4d-13b4-4ef5-a590-7a6bd93d072c_isrForCrosstalkSources_855810_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:17 proc_lsst.multi:146 [INFO]  found job 16966502 in provider astro
proc_lsst.multi INFO: found job 16966502 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b172c9c9d86f11e2ec78312d5251914d with result from task 291
parsl.dataflow.dflow DEBUG: Task 291 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/4112f4d4-0b7a-480c-bcb2-9a3dbff375df_isrForCrosstalkSources_855808_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/4112f4d4-0b7a-480c-bcb2-9a3dbff375df_isrForCrosstalkSources_855808_30.stderr
parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5dd6bf37cec3cc16f305213f491a6a30 with result from task 293
parsl.dataflow.dflow DEBUG: Task 293 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/fe94f797-3dc1-4e0d-b175-785e7a65adbb_isrForCrosstalkSources_855808_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/fe94f797-3dc1-4e0d-b175-785e7a65adbb_isrForCrosstalkSources_855808_60.stderr
parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7725e53ca0a7f5c52e57aa84d26896fe with result from task 294
parsl.dataflow.dflow DEBUG: Task 294 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/970352a1-db6f-4743-8e61-f2a6320f5427_isrForCrosstalkSources_855808_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/970352a1-db6f-4743-8e61-f2a6320f5427_isrForCrosstalkSources_855808_47.stderr
parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c7068f0924c5bc2614a93193457c822e with result from task 295
parsl.dataflow.dflow DEBUG: Task 295 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/1bd4a503-42f4-48b2-94c7-f31be150e0cf_isrForCrosstalkSources_855810_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/1bd4a503-42f4-48b2-94c7-f31be150e0cf_isrForCrosstalkSources_855810_47.stderr
parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 43b74b0242bb6c62bc8c925fb37e6028 with result from task 292
parsl.dataflow.dflow DEBUG: Task 292 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/bd578b3a-3eb4-4c6f-8eb3-f7d9a4d992b7_isrForCrosstalkSources_855718_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/bd578b3a-3eb4-4c6f-8eb3-f7d9a4d992b7_isrForCrosstalkSources_855718_42.stderr
parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5ea411fafa84f1d68f891ee4ff41ef86 with result from task 298
parsl.dataflow.dflow DEBUG: Task 298 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/5eeb69a9-688e-4fb6-a71c-71d50bf318b9_isrForCrosstalkSources_855810_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/5eeb69a9-688e-4fb6-a71c-71d50bf318b9_isrForCrosstalkSources_855810_33.stderr
parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8f8a1ad88489369dff8628bfc1631886 with result from task 297
parsl.dataflow.dflow DEBUG: Task 297 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/66160643-0138-49e2-8ce3-cc3efb24874b_isrForCrosstalkSources_855809_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/66160643-0138-49e2-8ce3-cc3efb24874b_isrForCrosstalkSources_855809_49.stderr
parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 99ecfb29ff01b5cba1bb06079ae42f4d with result from task 296
parsl.dataflow.dflow DEBUG: Task 296 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/dacfdb29-42b4-4e0d-884f-31a670bfa5cb_isrForCrosstalkSources_855810_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/dacfdb29-42b4-4e0d-884f-31a670bfa5cb_isrForCrosstalkSources_855810_19.stderr
parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 719d839d94a2c5eef47a82ebc1e6710a with result from task 240
parsl.dataflow.dflow DEBUG: Task 240 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/95c8204b-15fd-443b-b282-152268cd731b_isrForCrosstalkSources_855718_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/95c8204b-15fd-443b-b282-152268cd731b_isrForCrosstalkSources_855718_22.stderr
parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c0e5d7573887460b9a62b332a02cd7fe with result from task 236
parsl.dataflow.dflow DEBUG: Task 236 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/cdde492a-f1be-419c-b348-13e4d7a1bfe9_isrForCrosstalkSources_855810_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/cdde492a-f1be-419c-b348-13e4d7a1bfe9_isrForCrosstalkSources_855810_11.stderr
parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 997ff01a901915ce810dea8ad93798bb with result from task 241
parsl.dataflow.dflow DEBUG: Task 241 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/ffd28581-3f9d-4e6e-a69c-7921dd8a0973_isrForCrosstalkSources_855716_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/ffd28581-3f9d-4e6e-a69c-7921dd8a0973_isrForCrosstalkSources_855716_19.stderr
parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2b7ae5ae7adf7d7be0cbe7557533fac3 with result from task 234
parsl.dataflow.dflow DEBUG: Task 234 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/2c3f4669-3844-4792-8bc4-f330aaba005b_isrForCrosstalkSources_855808_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/2c3f4669-3844-4792-8bc4-f330aaba005b_isrForCrosstalkSources_855808_2.stderr
parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 02ed4025361fa8696705c4e271caf2f0 with result from task 235
parsl.dataflow.dflow DEBUG: Task 235 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/e67a975e-d97a-45b6-b542-4b0fc1b24f0d_isrForCrosstalkSources_855716_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/e67a975e-d97a-45b6-b542-4b0fc1b24f0d_isrForCrosstalkSources_855716_18.stderr
parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 128788b48e78e7f542b70caa6743421f with result from task 237
parsl.dataflow.dflow DEBUG: Task 237 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/9f09e316-f43a-4fff-be18-52b8c2314ce7_isrForCrosstalkSources_855809_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/9f09e316-f43a-4fff-be18-52b8c2314ce7_isrForCrosstalkSources_855809_6.stderr
parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 139a0e032e86b6a7b84c6200bfe1412a with result from task 238
parsl.dataflow.dflow DEBUG: Task 238 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/e6990b85-d9cd-450f-8f43-558de96da459_isrForCrosstalkSources_855718_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/e6990b85-d9cd-450f-8f43-558de96da459_isrForCrosstalkSources_855718_60.stderr
parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0fa7501f495a7f37abf134cd9441c4aa with result from task 239
parsl.dataflow.dflow DEBUG: Task 239 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/d44ac05f-a09b-4da1-a689-51c5a8a41fcd_isrForCrosstalkSources_855809_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/d44ac05f-a09b-4da1-a689-51c5a8a41fcd_isrForCrosstalkSources_855809_13.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:17 proc_lsst.multi:146 [INFO]  found job 16966503 in provider astro
proc_lsst.multi INFO: found job 16966503 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:17 proc_lsst.multi:146 [INFO]  found job 16966504 in provider astro
proc_lsst.multi INFO: found job 16966504 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:17 proc_lsst.multi:146 [INFO]  found job 16966505 in provider astro
proc_lsst.multi INFO: found job 16966505 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:18 proc_lsst.multi:146 [INFO]  found job 16966506 in provider astro
proc_lsst.multi INFO: found job 16966506 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:18 proc_lsst.multi:146 [INFO]  found job 16966507 in provider astro
proc_lsst.multi INFO: found job 16966507 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:18 proc_lsst.multi:146 [INFO]  found job 16966508 in provider astro
proc_lsst.multi INFO: found job 16966508 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:18 proc_lsst.multi:146 [INFO]  found job 16966509 in provider astro
proc_lsst.multi INFO: found job 16966509 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:18 proc_lsst.multi:146 [INFO]  found job 16966510 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966510 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966510,16966511,16966512,16966513,16966514'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966511 R
16966512 R
16966513 R
16966514 R
16966510 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966511 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966512 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966513 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966514 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966510 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:19 proc_lsst.multi:146 [INFO]  found job 16966511 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966511 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966510,16966511,16966512,16966513,16966514'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966511 R
16966512 R
16966513 R
16966514 R
16966510 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966511 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966512 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966513 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966514 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966510 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:19 proc_lsst.multi:146 [INFO]  found job 16966512 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966512 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966510,16966511,16966512,16966513,16966514'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966511 R
16966512 R
16966513 R
16966514 R
16966510 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966511 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966512 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966513 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966514 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966510 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:19 proc_lsst.multi:146 [INFO]  found job 16966513 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966513 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966510,16966511,16966512,16966513,16966514'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966511 R
16966512 R
16966513 R
16966514 R
16966510 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966511 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966512 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966513 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966514 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966510 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:19 proc_lsst.multi:146 [INFO]  found job 16966514 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966514 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966510,16966511,16966512,16966513,16966514'
parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7cd970cd0f6c298c674d905b9cdc56fa with result from task 338
parsl.dataflow.dflow DEBUG: Task 338 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/afa2ecb7-2d4e-4472-9cf6-4d26481ee39c_isrForCrosstalkSources_855717_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/afa2ecb7-2d4e-4472-9cf6-4d26481ee39c_isrForCrosstalkSources_855717_45.stderr
parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c19d65c0d35bcf3cfcc91090fb5e977f with result from task 327
parsl.dataflow.dflow DEBUG: Task 327 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/9fe358c8-5f59-4bdf-9abb-9096c9bff808_isrForCrosstalkSources_855809_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/9fe358c8-5f59-4bdf-9abb-9096c9bff808_isrForCrosstalkSources_855809_28.stderr
parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 765dbad89c33b35993e25db1c4986696 with result from task 331
parsl.dataflow.dflow DEBUG: Task 331 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/98ed8f42-fb13-4e29-a242-bf6ef2d195d1_isrForCrosstalkSources_855718_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/98ed8f42-fb13-4e29-a242-bf6ef2d195d1_isrForCrosstalkSources_855718_35.stderr
parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0fb31f04fb8028abfbfd4a51b16cc153 with result from task 340
parsl.dataflow.dflow DEBUG: Task 340 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/a69a3a20-2f63-4b19-9b2d-e4293df9c66a_isrForCrosstalkSources_855717_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/a69a3a20-2f63-4b19-9b2d-e4293df9c66a_isrForCrosstalkSources_855717_49.stderr
parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a3bac18e1b49dd38fcecbf283998b98a with result from task 339
parsl.providers.slurm.slurm DEBUG: squeue returned 16966511 R
16966512 R
16966513 R
16966514 R
16966510 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966511 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow DEBUG: Task 339 checkpointed
parsl.providers.slurm.slurm DEBUG: Updating job 16966512 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966513 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.providers.slurm.slurm DEBUG: Updating job 16966514 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/8961446a-2da1-4d67-9d69-db456295ac01_isrForCrosstalkSources_855718_5.stdout
parsl.providers.slurm.slurm DEBUG: Updating job 16966510 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:19 proc_lsst.multi:146 [INFO]  found job 16966515 in provider escience_a40
parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/8961446a-2da1-4d67-9d69-db456295ac01_isrForCrosstalkSources_855718_5.stderr
proc_lsst.multi INFO: found job 16966515 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 216e36229b5f317354937290483e3ceb with result from task 325
parsl.dataflow.dflow DEBUG: Task 325 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/c0524524-11aa-4621-adf6-4db829380ddc_isrForCrosstalkSources_855716_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/c0524524-11aa-4621-adf6-4db829380ddc_isrForCrosstalkSources_855716_45.stderr
parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0cab8ccacc3380fea1db13311a1ed5aa with result from task 328
parsl.dataflow.dflow DEBUG: Task 328 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/7d18c17b-c097-4f3f-bd7e-045c573fb50f_isrForCrosstalkSources_855717_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/7d18c17b-c097-4f3f-bd7e-045c573fb50f_isrForCrosstalkSources_855717_8.stderr
parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 23a9dc1f2503ddc22204b6df2fe1abd4 with result from task 329
parsl.dataflow.dflow DEBUG: Task 329 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/54c7c32a-b1af-4005-a831-87e7c65c6ef8_isrForCrosstalkSources_855716_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/54c7c32a-b1af-4005-a831-87e7c65c6ef8_isrForCrosstalkSources_855716_48.stderr
parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8cbae8c56753c481ce57e537c51e4307 with result from task 326
parsl.dataflow.dflow DEBUG: Task 326 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/54a8b699-6024-447c-9425-c49e932d6b2e_isrForCrosstalkSources_855808_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/54a8b699-6024-447c-9425-c49e932d6b2e_isrForCrosstalkSources_855808_12.stderr
parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 59a816459a4ed6dee1e6c96f41c96821 with result from task 330
parsl.dataflow.dflow DEBUG: Task 330 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/0af08179-9689-47bc-ad9e-b0a238bc6156_isrForCrosstalkSources_855809_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/0af08179-9689-47bc-ad9e-b0a238bc6156_isrForCrosstalkSources_855809_55.stderr
parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b76153123fb2f3952af9836bd24edd02 with result from task 332
parsl.dataflow.dflow DEBUG: Task 332 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/8178416b-6755-4d7a-ae41-119b57707b54_isrForCrosstalkSources_855717_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/8178416b-6755-4d7a-ae41-119b57707b54_isrForCrosstalkSources_855717_21.stderr
parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 450827e055b3bad8812220cd0e183761 with result from task 337
parsl.dataflow.dflow DEBUG: Task 337 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/1c344fc0-d9d8-4266-a857-5dbfd05a9c13_isrForCrosstalkSources_855718_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/1c344fc0-d9d8-4266-a857-5dbfd05a9c13_isrForCrosstalkSources_855718_51.stderr
parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 996e926cd5bc6f6dcc080d2803b255bc with result from task 341
parsl.dataflow.dflow DEBUG: Task 341 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/b3487d17-c7ed-4abf-820e-47da32589139_isrForCrosstalkSources_855808_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/b3487d17-c7ed-4abf-820e-47da32589139_isrForCrosstalkSources_855808_26.stderr
parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 975e9441dc93d3db44bf5b569f89c951 with result from task 335
parsl.dataflow.dflow DEBUG: Task 335 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/77523463-4689-4989-ae02-4116d0237ae4_isrForCrosstalkSources_855808_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/77523463-4689-4989-ae02-4116d0237ae4_isrForCrosstalkSources_855808_1.stderr
parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cef8ce6e2726bdc2933da2b372202733 with result from task 336
parsl.dataflow.dflow DEBUG: Task 336 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/0a1b3ab7-9bb5-4cc5-8e04-46e1d3e5f122_isrForCrosstalkSources_855716_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/0a1b3ab7-9bb5-4cc5-8e04-46e1d3e5f122_isrForCrosstalkSources_855716_36.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:19 proc_lsst.multi:146 [INFO]  found job 16966516 in provider escience_a40
proc_lsst.multi INFO: found job 16966516 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 68fbb605c571f9161fa6e7fd3ba4dc64 with result from task 334
parsl.dataflow.dflow DEBUG: Task 334 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/a5e030fd-6c5a-409a-a1a5-aecda345aa4d_isrForCrosstalkSources_855808_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/a5e030fd-6c5a-409a-a1a5-aecda345aa4d_isrForCrosstalkSources_855808_44.stderr
parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e617ab8768fe9b74126c862651c66325 with result from task 363
parsl.dataflow.dflow DEBUG: Task 363 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/0d6ed138-aad3-421c-85f6-388d9520e6fb_isrForCrosstalkSources_855809_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855809/0d6ed138-aad3-421c-85f6-388d9520e6fb_isrForCrosstalkSources_855809_9.stderr
parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fdc42f7a0b7786c24d6cc3b04abde9a7 with result from task 361
parsl.dataflow.dflow DEBUG: Task 361 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/c2a5b9aa-6faa-494d-8637-9905a6f1bd3e_isrForCrosstalkSources_855717_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/c2a5b9aa-6faa-494d-8637-9905a6f1bd3e_isrForCrosstalkSources_855717_56.stderr
parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e94af7c5738243b26b43c6ff4578aa0e with result from task 366
parsl.dataflow.dflow DEBUG: Task 366 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/9d407bd9-c54c-433d-af28-e7382a7e6dd8_isrForCrosstalkSources_855718_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855718/9d407bd9-c54c-433d-af28-e7382a7e6dd8_isrForCrosstalkSources_855718_19.stderr
parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 627dde1b6fbb05734bf50a6749778596 with result from task 367
parsl.dataflow.dflow DEBUG: Task 367 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/ffd0b4a8-c42b-4f99-a3aa-acc3b2fa3ae1_isrForCrosstalkSources_855808_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855808/ffd0b4a8-c42b-4f99-a3aa-acc3b2fa3ae1_isrForCrosstalkSources_855808_13.stderr
parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 13eac902a2200f02088cd30c62c89064 with result from task 364
parsl.dataflow.dflow DEBUG: Task 364 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/1ce2a30f-065e-4dc7-9880-f199c70511a3_isrForCrosstalkSources_855717_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/1ce2a30f-065e-4dc7-9880-f199c70511a3_isrForCrosstalkSources_855717_23.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:20 proc_lsst.multi:146 [INFO]  found job 16966517 in provider escience_a40
proc_lsst.multi INFO: found job 16966517 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4aea1a27934ab381d50368d9312c942d with result from task 360
parsl.dataflow.dflow DEBUG: Task 360 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/46c95bff-60e2-4165-b13b-52b8a0225c95_isrForCrosstalkSources_855810_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855810/46c95bff-60e2-4165-b13b-52b8a0225c95_isrForCrosstalkSources_855810_36.stderr
parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ed1c283d3f9f8cd204c0c488e045b39f with result from task 365
parsl.dataflow.dflow DEBUG: Task 365 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/08b3b017-c3fe-448a-b228-d85a275c6394_isrForCrosstalkSources_855716_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855716/08b3b017-c3fe-448a-b228-d85a275c6394_isrForCrosstalkSources_855716_56.stderr
parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 55bc755c842d1a71ce00f8b6e9b5f0dd with result from task 362
parsl.dataflow.dflow DEBUG: Task 362 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/e3eabb20-2264-48e8-8511-479d7c725865_isrForCrosstalkSources_855717_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/e3eabb20-2264-48e8-8511-479d7c725865_isrForCrosstalkSources_855717_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:20 proc_lsst.multi:146 [INFO]  found job 16966518 in provider escience_a40
proc_lsst.multi INFO: found job 16966518 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:20 proc_lsst.multi:146 [INFO]  found job 16966519 in provider escience_a40
proc_lsst.multi INFO: found job 16966519 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:20 proc_lsst.multi:146 [INFO]  found job 16966520 in provider escience_a40
proc_lsst.multi INFO: found job 16966520 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:20 proc_lsst.multi:146 [INFO]  found job 16966521 in provider escience_a40
proc_lsst.multi INFO: found job 16966521 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:21 proc_lsst.multi:146 [INFO]  found job 16966522 in provider escience_a40
proc_lsst.multi INFO: found job 16966522 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:23 proc_lsst.multi:146 [INFO]  found job 16966523 in provider escience_a40
proc_lsst.multi INFO: found job 16966523 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:23 proc_lsst.multi:146 [INFO]  found job 16966524 in provider escience_a40
proc_lsst.multi INFO: found job 16966524 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:23 proc_lsst.multi:146 [INFO]  found job 16966525 in provider escience_a40
proc_lsst.multi INFO: found job 16966525 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:24 proc_lsst.multi:146 [INFO]  found job 16966526 in provider escience_a40
proc_lsst.multi INFO: found job 16966526 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:24 proc_lsst.multi:146 [INFO]  found job 16966527 in provider escience_a40
proc_lsst.multi INFO: found job 16966527 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:24 proc_lsst.multi:146 [INFO]  found job 16966528 in provider escience_a40
proc_lsst.multi INFO: found job 16966528 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:24 proc_lsst.multi:146 [INFO]  found job 16966529 in provider escience_a40
proc_lsst.multi INFO: found job 16966529 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:24 proc_lsst.multi:146 [INFO]  found job 16966530 in provider escience_a40
proc_lsst.multi INFO: found job 16966530 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:25 proc_lsst.multi:146 [INFO]  found job 16966531 in provider escience_a40
proc_lsst.multi INFO: found job 16966531 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:25 proc_lsst.multi:146 [INFO]  found job 16966532 in provider escience_a40
proc_lsst.multi INFO: found job 16966532 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:25 proc_lsst.multi:146 [INFO]  found job 16966533 in provider escience_a40
proc_lsst.multi INFO: found job 16966533 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:25 proc_lsst.multi:146 [INFO]  found job 16966534 in provider escience_a40
proc_lsst.multi INFO: found job 16966534 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:26 proc_lsst.multi:146 [INFO]  found job 16966535 in provider checkpoint
proc_lsst.multi INFO: found job 16966535 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:26 proc_lsst.multi:146 [INFO]  found job 16966536 in provider checkpoint
proc_lsst.multi INFO: found job 16966536 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:26 proc_lsst.multi:146 [INFO]  found job 16966537 in provider checkpoint
proc_lsst.multi INFO: found job 16966537 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:26 proc_lsst.multi:146 [INFO]  found job 16966538 in provider checkpoint
proc_lsst.multi INFO: found job 16966538 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:26 proc_lsst.multi:146 [INFO]  found job 16966539 in provider checkpoint
proc_lsst.multi INFO: found job 16966539 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:27 proc_lsst.multi:146 [INFO]  found job 16966540 in provider checkpoint
proc_lsst.multi INFO: found job 16966540 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:27 proc_lsst.multi:146 [INFO]  found job 16966541 in provider checkpoint
proc_lsst.multi INFO: found job 16966541 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:27 proc_lsst.multi:146 [INFO]  found job 16966542 in provider checkpoint
proc_lsst.multi INFO: found job 16966542 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:27 proc_lsst.multi:146 [INFO]  found job 16966543 in provider checkpoint
proc_lsst.multi INFO: found job 16966543 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:27 proc_lsst.multi:146 [INFO]  found job 16966544 in provider checkpoint
proc_lsst.multi INFO: found job 16966544 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:28 proc_lsst.multi:146 [INFO]  found job 16966545 in provider checkpoint
proc_lsst.multi INFO: found job 16966545 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:28 proc_lsst.multi:146 [INFO]  found job 16966546 in provider checkpoint
proc_lsst.multi INFO: found job 16966546 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:28 proc_lsst.multi:146 [INFO]  found job 16966547 in provider checkpoint
proc_lsst.multi INFO: found job 16966547 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:28 proc_lsst.multi:146 [INFO]  found job 16966548 in provider checkpoint
proc_lsst.multi INFO: found job 16966548 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:28 proc_lsst.multi:146 [INFO]  found job 16966549 in provider checkpoint
proc_lsst.multi INFO: found job 16966549 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:29 proc_lsst.multi:146 [INFO]  found job 16966550 in provider checkpoint
proc_lsst.multi INFO: found job 16966550 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:29 proc_lsst.multi:146 [INFO]  found job 16966551 in provider checkpoint
proc_lsst.multi INFO: found job 16966551 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:29 proc_lsst.multi:146 [INFO]  found job 16966553 in provider checkpoint
proc_lsst.multi INFO: found job 16966553 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:29 proc_lsst.multi:146 [INFO]  found job 16966554 in provider checkpoint
proc_lsst.multi INFO: found job 16966554 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:29 proc_lsst.multi:146 [INFO]  found job 16966555 in provider checkpoint
proc_lsst.multi INFO: found job 16966555 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:30 proc_lsst.multi:146 [INFO]  found job 16966556 in provider checkpoint
proc_lsst.multi INFO: found job 16966556 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:30 proc_lsst.multi:146 [INFO]  found job 16966557 in provider checkpoint
proc_lsst.multi INFO: found job 16966557 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:30 proc_lsst.multi:146 [INFO]  found job 16966558 in provider checkpoint
proc_lsst.multi INFO: found job 16966558 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:30 proc_lsst.multi:146 [INFO]  found job 16966559 in provider checkpoint
proc_lsst.multi INFO: found job 16966559 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:31 proc_lsst.multi:146 [INFO]  found job 16966560 in provider checkpoint
proc_lsst.multi INFO: found job 16966560 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:31 proc_lsst.multi:146 [INFO]  found job 16966561 in provider checkpoint
proc_lsst.multi INFO: found job 16966561 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:31 proc_lsst.multi:146 [INFO]  found job 16966562 in provider checkpoint
proc_lsst.multi INFO: found job 16966562 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:31 proc_lsst.multi:146 [INFO]  found job 16966563 in provider checkpoint
proc_lsst.multi INFO: found job 16966563 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:32 proc_lsst.multi:146 [INFO]  found job 16966564 in provider checkpoint
proc_lsst.multi INFO: found job 16966564 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:32 proc_lsst.multi:146 [INFO]  found job 16966565 in provider checkpoint
proc_lsst.multi INFO: found job 16966565 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:32 proc_lsst.multi:146 [INFO]  found job 16966566 in provider checkpoint
proc_lsst.multi INFO: found job 16966566 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:32 proc_lsst.multi:146 [INFO]  found job 16966567 in provider checkpoint
proc_lsst.multi INFO: found job 16966567 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:32 proc_lsst.multi:146 [INFO]  found job 16966568 in provider checkpoint
proc_lsst.multi INFO: found job 16966568 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:33 proc_lsst.multi:146 [INFO]  found job 16966569 in provider checkpoint
proc_lsst.multi INFO: found job 16966569 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:33 proc_lsst.multi:146 [INFO]  found job 16966570 in provider checkpoint
proc_lsst.multi INFO: found job 16966570 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:33 proc_lsst.multi:146 [INFO]  found job 16966571 in provider checkpoint
proc_lsst.multi INFO: found job 16966571 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:33 proc_lsst.multi:146 [INFO]  found job 16966572 in provider checkpoint
proc_lsst.multi INFO: found job 16966572 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23383474330768
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23383474330768
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23383474330768
2024-03-09 17:38:43 proc_lsst.multi:146 [INFO]  found job 64340 in provider local
proc_lsst.multi INFO: found job 64340 in provider local
2024-03-09 17:38:43 proc_lsst.multi:146 [INFO]  found job 16966475 in provider astro
proc_lsst.multi INFO: found job 16966475 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:44 proc_lsst.multi:146 [INFO]  found job 16966476 in provider astro
proc_lsst.multi INFO: found job 16966476 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:44 proc_lsst.multi:146 [INFO]  found job 16966477 in provider astro
proc_lsst.multi INFO: found job 16966477 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:44 proc_lsst.multi:146 [INFO]  found job 16966478 in provider astro
proc_lsst.multi INFO: found job 16966478 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:44 proc_lsst.multi:146 [INFO]  found job 16966479 in provider astro
proc_lsst.multi INFO: found job 16966479 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:44 proc_lsst.multi:146 [INFO]  found job 16966480 in provider astro
proc_lsst.multi INFO: found job 16966480 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:45 proc_lsst.multi:146 [INFO]  found job 16966481 in provider astro
proc_lsst.multi INFO: found job 16966481 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:45 proc_lsst.multi:146 [INFO]  found job 16966482 in provider astro
proc_lsst.multi INFO: found job 16966482 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:45 proc_lsst.multi:146 [INFO]  found job 16966483 in provider astro
proc_lsst.multi INFO: found job 16966483 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:45 proc_lsst.multi:146 [INFO]  found job 16966484 in provider astro
proc_lsst.multi INFO: found job 16966484 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:45 proc_lsst.multi:146 [INFO]  found job 16966485 in provider astro
proc_lsst.multi INFO: found job 16966485 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:46 proc_lsst.multi:146 [INFO]  found job 16966486 in provider astro
proc_lsst.multi INFO: found job 16966486 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:46 proc_lsst.multi:146 [INFO]  found job 16966487 in provider astro
proc_lsst.multi INFO: found job 16966487 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:46 proc_lsst.multi:146 [INFO]  found job 16966488 in provider astro
proc_lsst.multi INFO: found job 16966488 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:46 proc_lsst.multi:146 [INFO]  found job 16966489 in provider astro
proc_lsst.multi INFO: found job 16966489 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:46 proc_lsst.multi:146 [INFO]  found job 16966490 in provider astro
proc_lsst.multi INFO: found job 16966490 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:47 proc_lsst.multi:146 [INFO]  found job 16966491 in provider astro
proc_lsst.multi INFO: found job 16966491 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:47 proc_lsst.multi:146 [INFO]  found job 16966492 in provider astro
proc_lsst.multi INFO: found job 16966492 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:47 proc_lsst.multi:146 [INFO]  found job 16966493 in provider astro
proc_lsst.multi INFO: found job 16966493 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:47 proc_lsst.multi:146 [INFO]  found job 16966499 in provider astro
proc_lsst.multi INFO: found job 16966499 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:47 proc_lsst.multi:146 [INFO]  found job 16966500 in provider astro
proc_lsst.multi INFO: found job 16966500 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:48 proc_lsst.multi:146 [INFO]  found job 16966501 in provider astro
proc_lsst.multi INFO: found job 16966501 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:48 proc_lsst.multi:146 [INFO]  found job 16966502 in provider astro
proc_lsst.multi INFO: found job 16966502 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:48 proc_lsst.multi:146 [INFO]  found job 16966503 in provider astro
proc_lsst.multi INFO: found job 16966503 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:48 proc_lsst.multi:146 [INFO]  found job 16966504 in provider astro
proc_lsst.multi INFO: found job 16966504 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:48 proc_lsst.multi:146 [INFO]  found job 16966505 in provider astro
proc_lsst.multi INFO: found job 16966505 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:49 proc_lsst.multi:146 [INFO]  found job 16966506 in provider astro
proc_lsst.multi INFO: found job 16966506 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:49 proc_lsst.multi:146 [INFO]  found job 16966507 in provider astro
proc_lsst.multi INFO: found job 16966507 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:49 proc_lsst.multi:146 [INFO]  found job 16966508 in provider astro
proc_lsst.multi INFO: found job 16966508 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:49 proc_lsst.multi:146 [INFO]  found job 16966509 in provider astro
proc_lsst.multi INFO: found job 16966509 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966475,16966476,16966477,16966478,16966479,16966480,16966481,16966482,16966483,16966484,16966485,16966486,16966487,16966488,16966489,16966490,16966491,16966492,16966493,16966499,16966500,16966501,16966502,16966503,16966504,16966505,16966506,16966507,16966508,16966509'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966505 R
16966506 R
16966507 R
16966508 R
16966509 R
16966500 R
16966501 R
16966502 R
16966503 R
16966504 R
16966499 R
16966485 R
16966486 R
16966487 R
16966488 R
16966489 R
16966490 R
16966491 R
16966492 R
16966493 R
16966482 R
16966483 R
16966484 R
16966481 R
16966480 R
16966475 R
16966476 R
16966477 R
16966478 R
16966479 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966505 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966506 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966507 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966508 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966509 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966503 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966504 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966491 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966492 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966479 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:49 proc_lsst.multi:146 [INFO]  found job 16966510 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966510 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966510,16966511,16966512,16966513,16966514'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966511 R
16966512 R
16966513 R
16966514 R
16966510 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966511 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966512 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966513 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966514 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966510 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:50 proc_lsst.multi:146 [INFO]  found job 16966511 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966511 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966510,16966511,16966512,16966513,16966514'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966511 R
16966512 R
16966513 R
16966514 R
16966510 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966511 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966512 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966513 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966514 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966510 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:50 proc_lsst.multi:146 [INFO]  found job 16966512 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966512 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966510,16966511,16966512,16966513,16966514'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966511 R
16966512 R
16966513 R
16966514 R
16966510 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966511 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966512 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966513 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966514 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966510 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:50 proc_lsst.multi:146 [INFO]  found job 16966513 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966513 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966510,16966511,16966512,16966513,16966514'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966511 R
16966512 R
16966513 R
16966514 R
16966510 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966511 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966512 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966513 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966514 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966510 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:50 proc_lsst.multi:146 [INFO]  found job 16966514 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966514 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966510,16966511,16966512,16966513,16966514'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966511 R
16966512 R
16966513 R
16966514 R
16966510 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966511 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966512 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966513 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966514 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966510 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:50 proc_lsst.multi:146 [INFO]  found job 16966515 in provider escience_a40
proc_lsst.multi INFO: found job 16966515 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:51 proc_lsst.multi:146 [INFO]  found job 16966516 in provider escience_a40
proc_lsst.multi INFO: found job 16966516 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:51 proc_lsst.multi:146 [INFO]  found job 16966517 in provider escience_a40
proc_lsst.multi INFO: found job 16966517 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:51 proc_lsst.multi:146 [INFO]  found job 16966518 in provider escience_a40
proc_lsst.multi INFO: found job 16966518 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:51 proc_lsst.multi:146 [INFO]  found job 16966519 in provider escience_a40
proc_lsst.multi INFO: found job 16966519 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:51 proc_lsst.multi:146 [INFO]  found job 16966520 in provider escience_a40
proc_lsst.multi INFO: found job 16966520 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:52 proc_lsst.multi:146 [INFO]  found job 16966521 in provider escience_a40
proc_lsst.multi INFO: found job 16966521 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:52 proc_lsst.multi:146 [INFO]  found job 16966522 in provider escience_a40
proc_lsst.multi INFO: found job 16966522 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:52 proc_lsst.multi:146 [INFO]  found job 16966523 in provider escience_a40
proc_lsst.multi INFO: found job 16966523 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:52 proc_lsst.multi:146 [INFO]  found job 16966524 in provider escience_a40
proc_lsst.multi INFO: found job 16966524 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:52 proc_lsst.multi:146 [INFO]  found job 16966525 in provider escience_a40
proc_lsst.multi INFO: found job 16966525 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:53 proc_lsst.multi:146 [INFO]  found job 16966526 in provider escience_a40
proc_lsst.multi INFO: found job 16966526 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:53 proc_lsst.multi:146 [INFO]  found job 16966527 in provider escience_a40
proc_lsst.multi INFO: found job 16966527 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:53 proc_lsst.multi:146 [INFO]  found job 16966528 in provider escience_a40
proc_lsst.multi INFO: found job 16966528 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:53 proc_lsst.multi:146 [INFO]  found job 16966529 in provider escience_a40
proc_lsst.multi INFO: found job 16966529 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:53 proc_lsst.multi:146 [INFO]  found job 16966530 in provider escience_a40
proc_lsst.multi INFO: found job 16966530 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:54 proc_lsst.multi:146 [INFO]  found job 16966531 in provider escience_a40
proc_lsst.multi INFO: found job 16966531 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry db743d22b6dc1a0d345009ca0e6c52d7 with result from task 69
parsl.dataflow.dflow INFO: DFK cleanup initiated
parsl.dataflow.dflow INFO: Summary of tasks in DFK:
parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0
parsl.dataflow.dflow INFO: Tasks in state States.pending: 0
parsl.dataflow.dflow INFO: Tasks in state States.running: 0
parsl.dataflow.dflow DEBUG: Task 69 checkpointed
parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 372
parsl.dataflow.dflow INFO: Tasks in state States.failed: 0
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0
parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/6aa09c6b-cc69-4f35-a4ba-0b1ff713ffd8_isrForCrosstalkSources_855717_15.stdout
parsl.dataflow.dflow INFO: Tasks in state States.launched: 0
parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/submit/DEEP/20190507/A0c/science#step0/20240309T173428Z/logs/isrForCrosstalkSources/855717/6aa09c6b-cc69-4f35-a4ba-0b1ff713ffd8_isrForCrosstalkSources_855717_15.stderr
parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0
parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0
parsl.dataflow.dflow INFO: Tasks in state States.joining: 0
parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0
parsl.dataflow.dflow INFO: End of summary
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Closing job status poller
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:56 proc_lsst.multi:146 [INFO]  found job 16966532 in provider escience_a40
proc_lsst.multi INFO: found job 16966532 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:56 proc_lsst.multi:146 [INFO]  found job 16966533 in provider escience_a40
proc_lsst.multi INFO: found job 16966533 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:56 proc_lsst.multi:146 [INFO]  found job 16966534 in provider escience_a40
proc_lsst.multi INFO: found job 16966534 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966515,16966516,16966517,16966518,16966519,16966520,16966521,16966522,16966523,16966524,16966525,16966526,16966527,16966528,16966529,16966530,16966531,16966532,16966533,16966534'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966532 R
16966533 R
16966534 R
16966528 R
16966529 R
16966530 R
16966531 R
16966523 R
16966524 R
16966525 R
16966526 R
16966527 R
16966521 R
16966522 R
16966519 R
16966520 R
16966517 R
16966518 R
16966515 R
16966516 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966532 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966533 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966534 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966528 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966529 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966530 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966531 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966523 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966524 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966525 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966526 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966527 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966521 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966522 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966519 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966520 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966517 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966518 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966515 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966516 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:57 proc_lsst.multi:146 [INFO]  found job 16966535 in provider checkpoint
proc_lsst.multi INFO: found job 16966535 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:57 proc_lsst.multi:146 [INFO]  found job 16966536 in provider checkpoint
proc_lsst.multi INFO: found job 16966536 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:57 proc_lsst.multi:146 [INFO]  found job 16966537 in provider checkpoint
proc_lsst.multi INFO: found job 16966537 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:58 proc_lsst.multi:146 [INFO]  found job 16966538 in provider checkpoint
proc_lsst.multi INFO: found job 16966538 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:58 proc_lsst.multi:146 [INFO]  found job 16966539 in provider checkpoint
proc_lsst.multi INFO: found job 16966539 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:58 proc_lsst.multi:146 [INFO]  found job 16966540 in provider checkpoint
proc_lsst.multi INFO: found job 16966540 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:58 proc_lsst.multi:146 [INFO]  found job 16966541 in provider checkpoint
proc_lsst.multi INFO: found job 16966541 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:58 proc_lsst.multi:146 [INFO]  found job 16966542 in provider checkpoint
proc_lsst.multi INFO: found job 16966542 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:59 proc_lsst.multi:146 [INFO]  found job 16966543 in provider checkpoint
proc_lsst.multi INFO: found job 16966543 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:59 proc_lsst.multi:146 [INFO]  found job 16966544 in provider checkpoint
proc_lsst.multi INFO: found job 16966544 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:59 proc_lsst.multi:146 [INFO]  found job 16966545 in provider checkpoint
proc_lsst.multi INFO: found job 16966545 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:59 proc_lsst.multi:146 [INFO]  found job 16966546 in provider checkpoint
proc_lsst.multi INFO: found job 16966546 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:38:59 proc_lsst.multi:146 [INFO]  found job 16966547 in provider checkpoint
proc_lsst.multi INFO: found job 16966547 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:00 proc_lsst.multi:146 [INFO]  found job 16966548 in provider checkpoint
proc_lsst.multi INFO: found job 16966548 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:00 proc_lsst.multi:146 [INFO]  found job 16966549 in provider checkpoint
proc_lsst.multi INFO: found job 16966549 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:00 proc_lsst.multi:146 [INFO]  found job 16966550 in provider checkpoint
proc_lsst.multi INFO: found job 16966550 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:00 proc_lsst.multi:146 [INFO]  found job 16966551 in provider checkpoint
proc_lsst.multi INFO: found job 16966551 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:00 proc_lsst.multi:146 [INFO]  found job 16966553 in provider checkpoint
proc_lsst.multi INFO: found job 16966553 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:01 proc_lsst.multi:146 [INFO]  found job 16966554 in provider checkpoint
proc_lsst.multi INFO: found job 16966554 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:01 proc_lsst.multi:146 [INFO]  found job 16966555 in provider checkpoint
proc_lsst.multi INFO: found job 16966555 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:01 proc_lsst.multi:146 [INFO]  found job 16966556 in provider checkpoint
proc_lsst.multi INFO: found job 16966556 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:01 proc_lsst.multi:146 [INFO]  found job 16966557 in provider checkpoint
proc_lsst.multi INFO: found job 16966557 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:02 proc_lsst.multi:146 [INFO]  found job 16966558 in provider checkpoint
proc_lsst.multi INFO: found job 16966558 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:02 proc_lsst.multi:146 [INFO]  found job 16966559 in provider checkpoint
proc_lsst.multi INFO: found job 16966559 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:02 proc_lsst.multi:146 [INFO]  found job 16966560 in provider checkpoint
proc_lsst.multi INFO: found job 16966560 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:02 proc_lsst.multi:146 [INFO]  found job 16966561 in provider checkpoint
proc_lsst.multi INFO: found job 16966561 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:02 proc_lsst.multi:146 [INFO]  found job 16966562 in provider checkpoint
proc_lsst.multi INFO: found job 16966562 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:03 proc_lsst.multi:146 [INFO]  found job 16966563 in provider checkpoint
proc_lsst.multi INFO: found job 16966563 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:03 proc_lsst.multi:146 [INFO]  found job 16966564 in provider checkpoint
proc_lsst.multi INFO: found job 16966564 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:03 proc_lsst.multi:146 [INFO]  found job 16966565 in provider checkpoint
proc_lsst.multi INFO: found job 16966565 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:03 proc_lsst.multi:146 [INFO]  found job 16966566 in provider checkpoint
proc_lsst.multi INFO: found job 16966566 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:03 proc_lsst.multi:146 [INFO]  found job 16966567 in provider checkpoint
proc_lsst.multi INFO: found job 16966567 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:04 proc_lsst.multi:146 [INFO]  found job 16966568 in provider checkpoint
proc_lsst.multi INFO: found job 16966568 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:04 proc_lsst.multi:146 [INFO]  found job 16966569 in provider checkpoint
proc_lsst.multi INFO: found job 16966569 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:04 proc_lsst.multi:146 [INFO]  found job 16966570 in provider checkpoint
proc_lsst.multi INFO: found job 16966570 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:04 proc_lsst.multi:146 [INFO]  found job 16966571 in provider checkpoint
proc_lsst.multi INFO: found job 16966571 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
2024-03-09 17:39:04 proc_lsst.multi:146 [INFO]  found job 16966572 in provider checkpoint
proc_lsst.multi INFO: found job 16966572 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16966535,16966536,16966537,16966538,16966539,16966540,16966541,16966542,16966543,16966544,16966545,16966546,16966547,16966548,16966549,16966550,16966551,16966553,16966554,16966555,16966556,16966557,16966558,16966559,16966560,16966561,16966562,16966563,16966564,16966565,16966566,16966567,16966568,16966569,16966570,16966571,16966572'
parsl.providers.slurm.slurm DEBUG: squeue returned 16966535 R
16966536 R
16966537 R
16966538 R
16966539 R
16966540 R
16966541 R
16966542 R
16966543 R
16966544 R
16966545 R
16966546 R
16966547 R
16966548 R
16966549 R
16966550 R
16966551 R
16966553 R
16966554 R
16966555 R
16966556 R
16966557 R
16966558 R
16966559 R
16966560 R
16966561 R
16966562 R
16966563 R
16966564 R
16966565 R
16966566 R
16966567 R
16966568 R
16966569 R
16966570 R
16966571 R
16966572 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16966535 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966536 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966537 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966538 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966539 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966540 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966541 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966542 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966543 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966544 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966545 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966546 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966547 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966548 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966549 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966550 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966551 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966553 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966554 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966555 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966556 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966557 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966558 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966559 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966560 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966561 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966562 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966563 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966564 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966565 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966566 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966567 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966568 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966569 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966570 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966571 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16966572 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 0
parsl.dataflow.strategy DEBUG: Executor multi has 0 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 1: Executor has no active tasks
parsl.dataflow.strategy DEBUG: Strategy case 1b: Executor has no active tasks, and more (93) than minimum blocks (0)
parsl.dataflow.strategy DEBUG: Starting idle timer for executor. If idle time exceeds 120.0s, blocks will be scaled in
parsl.dataflow.strategy DEBUG: Idle time 4.291534423828125e-06s is less than max_idletime 120.0s for executor multi; not scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23383474330768
parsl.dataflow.dflow INFO: Terminated job status poller
parsl.dataflow.dflow INFO: Scaling in and shutting down executors
parsl.dataflow.dflow INFO: Scaling in executor multi
parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=93, block_ids=[]
parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 93 blocks
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9d52bf148c09
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9d52bf148c09
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7990c9a59938
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7990c9a59938
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e01518898851
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e01518898851
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 939063fe05eb
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 939063fe05eb
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d07fe17a0a22
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d07fe17a0a22
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c509df7e1ca7
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c509df7e1ca7
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: feb6ece1d27d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: feb6ece1d27d
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9825c45c6570
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9825c45c6570
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b924e10abcd8
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b924e10abcd8
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6fec0756475f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6fec0756475f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fe98c8fd7473
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fe98c8fd7473
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 31dafea5e355
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 31dafea5e355
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c0e5b4107246
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c0e5b4107246
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 43e4dcd8efb1
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 43e4dcd8efb1
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a16012b8a866
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a16012b8a866
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f02208c71bd6
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f02208c71bd6
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 136839ac8dbe
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 136839ac8dbe
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4eef106a45a3
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4eef106a45a3
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a5b48bdbab61
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a5b48bdbab61
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 755dfd542deb
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 755dfd542deb
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c514e99e68d9
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c514e99e68d9
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4ab0ece7e276
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4ab0ece7e276
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 574fea9fb46d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 574fea9fb46d
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 012302fdc844
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 012302fdc844
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 69c6c2dda6db
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 69c6c2dda6db
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7d69e3e3af03
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7d69e3e3af03
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 84f4f56fd3a2
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 84f4f56fd3a2
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: aeb01ec8b73d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: aeb01ec8b73d
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 03bddc732243
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 03bddc732243
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 490d10519f35
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 490d10519f35
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b1aa79e54566
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b1aa79e54566
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 79a851fcd459
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 79a851fcd459
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 31f3f26ec0ff
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 31f3f26ec0ff
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0cf348bfbd27
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0cf348bfbd27
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 11652be37621
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 11652be37621
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1f778e8a6af5
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1f778e8a6af5
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f804ab6ccc11
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f804ab6ccc11
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 920ad79ec66c
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 920ad79ec66c
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3aa298ba627c
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3aa298ba627c
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: be7f4f2fd495
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: be7f4f2fd495
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a037083b0e7e
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a037083b0e7e
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 62b1f0b7f57a
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 62b1f0b7f57a
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5baebc09786c
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5baebc09786c
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 20819f2232a2
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 20819f2232a2
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: be3eee5762cf
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: be3eee5762cf
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a6cf3c9237fc
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a6cf3c9237fc
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 51613efe0f30
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 51613efe0f30
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c6cf236703a3
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c6cf236703a3
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 11abbe9d12d0
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 11abbe9d12d0
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 48b6f19802aa
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 48b6f19802aa
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 713a21199851
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 713a21199851
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 52bd80bf0061
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 52bd80bf0061
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 158170f809e1
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 158170f809e1
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c217af96c3ce
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c217af96c3ce
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e379c277f57a
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e379c277f57a
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: beebdfd58aa0
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: beebdfd58aa0
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 63856c0736bd
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 63856c0736bd
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 05676512ff69
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 05676512ff69
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8770a5665e85
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8770a5665e85
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6c7b087fdd4c
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6c7b087fdd4c
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: acca85dbb30b
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: acca85dbb30b
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2f286db44893
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2f286db44893
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d7fcb1d6da25
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d7fcb1d6da25
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f6c3470882e9
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f6c3470882e9
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 437f268787fd
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 437f268787fd
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 86f7c2824b61
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 86f7c2824b61
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3837fbba404f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3837fbba404f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b75c1dac8db1
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b75c1dac8db1
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1a562e4b8426
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1a562e4b8426
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a3c6015af368
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a3c6015af368
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b8eac2a9dfdd
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b8eac2a9dfdd
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2dcf242f930a
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2dcf242f930a
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8e8ddba4a236
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8e8ddba4a236
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a43fdd5ba60f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a43fdd5ba60f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7191d6777091
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7191d6777091
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f53021f29034
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f53021f29034
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 29fa3d936456
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 29fa3d936456
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3912f7e140ee
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3912f7e140ee
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0f0bdfe33b2f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0f0bdfe33b2f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 78d687853215
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 78d687853215
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 08a5ad33b8a5
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 08a5ad33b8a5
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f01e0a3a5fe8
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f01e0a3a5fe8
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: efc8455b0833
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: efc8455b0833
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fdbffd3c0d11
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fdbffd3c0d11
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 299f91d8a33a
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 299f91d8a33a
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9dee983a7dc2
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9dee983a7dc2
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 87018a492132
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 87018a492132
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 09605572515b
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 09605572515b
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4b3f020329f7
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4b3f020329f7
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0cb3d95b85c4
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0cb3d95b85c4
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 80c4b229e6fa
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 80c4b229e6fa
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 473f01411f86
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 473f01411f86
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 437d08a65ee6
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 437d08a65ee6
2024-03-09 17:39:05 proc_lsst.multi:146 [INFO]  found job 16966515 in provider escience_a40
proc_lsst.multi INFO: found job 16966515 in provider escience_a40
2024-03-09 17:39:05 proc_lsst.multi:201 [INFO]  cancelling 16966515 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966515 on provider escience_a40
2024-03-09 17:39:05 proc_lsst.multi:146 [INFO]  found job 16966545 in provider checkpoint
proc_lsst.multi INFO: found job 16966545 in provider checkpoint
2024-03-09 17:39:05 proc_lsst.multi:201 [INFO]  cancelling 16966545 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966545 on provider checkpoint
2024-03-09 17:39:05 proc_lsst.multi:146 [INFO]  found job 16966541 in provider checkpoint
proc_lsst.multi INFO: found job 16966541 in provider checkpoint
2024-03-09 17:39:05 proc_lsst.multi:201 [INFO]  cancelling 16966541 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966541 on provider checkpoint
2024-03-09 17:39:05 proc_lsst.multi:146 [INFO]  found job 16966549 in provider checkpoint
proc_lsst.multi INFO: found job 16966549 in provider checkpoint
2024-03-09 17:39:05 proc_lsst.multi:201 [INFO]  cancelling 16966549 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966549 on provider checkpoint
2024-03-09 17:39:05 proc_lsst.multi:146 [INFO]  found job 16966568 in provider checkpoint
proc_lsst.multi INFO: found job 16966568 in provider checkpoint
2024-03-09 17:39:05 proc_lsst.multi:201 [INFO]  cancelling 16966568 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966568 on provider checkpoint
2024-03-09 17:39:06 proc_lsst.multi:146 [INFO]  found job 16966548 in provider checkpoint
proc_lsst.multi INFO: found job 16966548 in provider checkpoint
2024-03-09 17:39:06 proc_lsst.multi:201 [INFO]  cancelling 16966548 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966548 on provider checkpoint
2024-03-09 17:39:06 proc_lsst.multi:146 [INFO]  found job 16966538 in provider checkpoint
proc_lsst.multi INFO: found job 16966538 in provider checkpoint
2024-03-09 17:39:06 proc_lsst.multi:201 [INFO]  cancelling 16966538 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966538 on provider checkpoint
2024-03-09 17:39:06 proc_lsst.multi:146 [INFO]  found job 16966562 in provider checkpoint
proc_lsst.multi INFO: found job 16966562 in provider checkpoint
2024-03-09 17:39:06 proc_lsst.multi:201 [INFO]  cancelling 16966562 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966562 on provider checkpoint
2024-03-09 17:39:06 proc_lsst.multi:146 [INFO]  found job 16966539 in provider checkpoint
proc_lsst.multi INFO: found job 16966539 in provider checkpoint
2024-03-09 17:39:06 proc_lsst.multi:201 [INFO]  cancelling 16966539 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966539 on provider checkpoint
2024-03-09 17:39:06 proc_lsst.multi:146 [INFO]  found job 16966540 in provider checkpoint
proc_lsst.multi INFO: found job 16966540 in provider checkpoint
2024-03-09 17:39:06 proc_lsst.multi:201 [INFO]  cancelling 16966540 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966540 on provider checkpoint
2024-03-09 17:39:06 proc_lsst.multi:146 [INFO]  found job 16966566 in provider checkpoint
proc_lsst.multi INFO: found job 16966566 in provider checkpoint
2024-03-09 17:39:06 proc_lsst.multi:201 [INFO]  cancelling 16966566 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966566 on provider checkpoint
2024-03-09 17:39:07 proc_lsst.multi:146 [INFO]  found job 16966536 in provider checkpoint
proc_lsst.multi INFO: found job 16966536 in provider checkpoint
2024-03-09 17:39:07 proc_lsst.multi:201 [INFO]  cancelling 16966536 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966536 on provider checkpoint
2024-03-09 17:39:07 proc_lsst.multi:146 [INFO]  found job 16966571 in provider checkpoint
proc_lsst.multi INFO: found job 16966571 in provider checkpoint
2024-03-09 17:39:07 proc_lsst.multi:201 [INFO]  cancelling 16966571 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966571 on provider checkpoint
2024-03-09 17:39:07 proc_lsst.multi:146 [INFO]  found job 16966535 in provider checkpoint
proc_lsst.multi INFO: found job 16966535 in provider checkpoint
2024-03-09 17:39:07 proc_lsst.multi:201 [INFO]  cancelling 16966535 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966535 on provider checkpoint
2024-03-09 17:39:07 proc_lsst.multi:146 [INFO]  found job 16966572 in provider checkpoint
proc_lsst.multi INFO: found job 16966572 in provider checkpoint
2024-03-09 17:39:07 proc_lsst.multi:201 [INFO]  cancelling 16966572 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966572 on provider checkpoint
2024-03-09 17:39:07 proc_lsst.multi:146 [INFO]  found job 16966561 in provider checkpoint
proc_lsst.multi INFO: found job 16966561 in provider checkpoint
2024-03-09 17:39:07 proc_lsst.multi:201 [INFO]  cancelling 16966561 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966561 on provider checkpoint
2024-03-09 17:39:07 proc_lsst.multi:146 [INFO]  found job 16966570 in provider checkpoint
proc_lsst.multi INFO: found job 16966570 in provider checkpoint
2024-03-09 17:39:07 proc_lsst.multi:201 [INFO]  cancelling 16966570 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966570 on provider checkpoint
2024-03-09 17:39:08 proc_lsst.multi:146 [INFO]  found job 16966569 in provider checkpoint
proc_lsst.multi INFO: found job 16966569 in provider checkpoint
2024-03-09 17:39:08 proc_lsst.multi:201 [INFO]  cancelling 16966569 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966569 on provider checkpoint
2024-03-09 17:39:08 proc_lsst.multi:146 [INFO]  found job 16966567 in provider checkpoint
proc_lsst.multi INFO: found job 16966567 in provider checkpoint
2024-03-09 17:39:08 proc_lsst.multi:201 [INFO]  cancelling 16966567 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966567 on provider checkpoint
2024-03-09 17:39:08 proc_lsst.multi:146 [INFO]  found job 16966564 in provider checkpoint
proc_lsst.multi INFO: found job 16966564 in provider checkpoint
2024-03-09 17:39:08 proc_lsst.multi:201 [INFO]  cancelling 16966564 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966564 on provider checkpoint
2024-03-09 17:39:08 proc_lsst.multi:146 [INFO]  found job 16966565 in provider checkpoint
proc_lsst.multi INFO: found job 16966565 in provider checkpoint
2024-03-09 17:39:08 proc_lsst.multi:201 [INFO]  cancelling 16966565 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966565 on provider checkpoint
2024-03-09 17:39:08 proc_lsst.multi:146 [INFO]  found job 16966563 in provider checkpoint
proc_lsst.multi INFO: found job 16966563 in provider checkpoint
2024-03-09 17:39:08 proc_lsst.multi:201 [INFO]  cancelling 16966563 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966563 on provider checkpoint
2024-03-09 17:39:08 proc_lsst.multi:146 [INFO]  found job 16966504 in provider astro
proc_lsst.multi INFO: found job 16966504 in provider astro
2024-03-09 17:39:08 proc_lsst.multi:201 [INFO]  cancelling 16966504 on provider astro
proc_lsst.multi INFO: cancelling 16966504 on provider astro
2024-03-09 17:39:09 proc_lsst.multi:146 [INFO]  found job 16966505 in provider astro
proc_lsst.multi INFO: found job 16966505 in provider astro
2024-03-09 17:39:09 proc_lsst.multi:201 [INFO]  cancelling 16966505 on provider astro
proc_lsst.multi INFO: cancelling 16966505 on provider astro
2024-03-09 17:39:09 proc_lsst.multi:146 [INFO]  found job 16966486 in provider astro
proc_lsst.multi INFO: found job 16966486 in provider astro
2024-03-09 17:39:09 proc_lsst.multi:201 [INFO]  cancelling 16966486 on provider astro
proc_lsst.multi INFO: cancelling 16966486 on provider astro
2024-03-09 17:39:09 proc_lsst.multi:146 [INFO]  found job 16966506 in provider astro
proc_lsst.multi INFO: found job 16966506 in provider astro
2024-03-09 17:39:09 proc_lsst.multi:201 [INFO]  cancelling 16966506 on provider astro
proc_lsst.multi INFO: cancelling 16966506 on provider astro
2024-03-09 17:39:09 proc_lsst.multi:146 [INFO]  found job 16966501 in provider astro
proc_lsst.multi INFO: found job 16966501 in provider astro
2024-03-09 17:39:09 proc_lsst.multi:201 [INFO]  cancelling 16966501 on provider astro
proc_lsst.multi INFO: cancelling 16966501 on provider astro
2024-03-09 17:39:09 proc_lsst.multi:146 [INFO]  found job 16966482 in provider astro
proc_lsst.multi INFO: found job 16966482 in provider astro
2024-03-09 17:39:09 proc_lsst.multi:201 [INFO]  cancelling 16966482 on provider astro
proc_lsst.multi INFO: cancelling 16966482 on provider astro
2024-03-09 17:39:09 proc_lsst.multi:146 [INFO]  found job 16966514 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966514 in provider escience_rtx6k
2024-03-09 17:39:09 proc_lsst.multi:201 [INFO]  cancelling 16966514 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16966514 on provider escience_rtx6k
2024-03-09 17:39:10 proc_lsst.multi:146 [INFO]  found job 16966490 in provider astro
proc_lsst.multi INFO: found job 16966490 in provider astro
2024-03-09 17:39:10 proc_lsst.multi:201 [INFO]  cancelling 16966490 on provider astro
proc_lsst.multi INFO: cancelling 16966490 on provider astro
2024-03-09 17:39:10 proc_lsst.multi:146 [INFO]  found job 16966521 in provider escience_a40
proc_lsst.multi INFO: found job 16966521 in provider escience_a40
2024-03-09 17:39:10 proc_lsst.multi:201 [INFO]  cancelling 16966521 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966521 on provider escience_a40
2024-03-09 17:39:10 proc_lsst.multi:146 [INFO]  found job 16966491 in provider astro
proc_lsst.multi INFO: found job 16966491 in provider astro
2024-03-09 17:39:10 proc_lsst.multi:201 [INFO]  cancelling 16966491 on provider astro
proc_lsst.multi INFO: cancelling 16966491 on provider astro
2024-03-09 17:39:10 proc_lsst.multi:146 [INFO]  found job 16966513 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966513 in provider escience_rtx6k
2024-03-09 17:39:10 proc_lsst.multi:201 [INFO]  cancelling 16966513 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16966513 on provider escience_rtx6k
2024-03-09 17:39:10 proc_lsst.multi:146 [INFO]  found job 16966516 in provider escience_a40
proc_lsst.multi INFO: found job 16966516 in provider escience_a40
2024-03-09 17:39:10 proc_lsst.multi:201 [INFO]  cancelling 16966516 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966516 on provider escience_a40
2024-03-09 17:39:10 proc_lsst.multi:146 [INFO]  found job 16966508 in provider astro
proc_lsst.multi INFO: found job 16966508 in provider astro
2024-03-09 17:39:10 proc_lsst.multi:201 [INFO]  cancelling 16966508 on provider astro
proc_lsst.multi INFO: cancelling 16966508 on provider astro
2024-03-09 17:39:11 proc_lsst.multi:146 [INFO]  found job 16966479 in provider astro
proc_lsst.multi INFO: found job 16966479 in provider astro
2024-03-09 17:39:11 proc_lsst.multi:201 [INFO]  cancelling 16966479 on provider astro
proc_lsst.multi INFO: cancelling 16966479 on provider astro
2024-03-09 17:39:11 proc_lsst.multi:146 [INFO]  found job 16966488 in provider astro
proc_lsst.multi INFO: found job 16966488 in provider astro
2024-03-09 17:39:11 proc_lsst.multi:201 [INFO]  cancelling 16966488 on provider astro
proc_lsst.multi INFO: cancelling 16966488 on provider astro
2024-03-09 17:39:11 proc_lsst.multi:146 [INFO]  found job 16966524 in provider escience_a40
proc_lsst.multi INFO: found job 16966524 in provider escience_a40
2024-03-09 17:39:11 proc_lsst.multi:201 [INFO]  cancelling 16966524 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966524 on provider escience_a40
2024-03-09 17:39:11 proc_lsst.multi:146 [INFO]  found job 16966483 in provider astro
proc_lsst.multi INFO: found job 16966483 in provider astro
2024-03-09 17:39:11 proc_lsst.multi:201 [INFO]  cancelling 16966483 on provider astro
proc_lsst.multi INFO: cancelling 16966483 on provider astro
2024-03-09 17:39:11 proc_lsst.multi:146 [INFO]  found job 16966522 in provider escience_a40
proc_lsst.multi INFO: found job 16966522 in provider escience_a40
2024-03-09 17:39:11 proc_lsst.multi:201 [INFO]  cancelling 16966522 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966522 on provider escience_a40
2024-03-09 17:39:11 proc_lsst.multi:146 [INFO]  found job 16966510 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966510 in provider escience_rtx6k
2024-03-09 17:39:11 proc_lsst.multi:201 [INFO]  cancelling 16966510 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16966510 on provider escience_rtx6k
2024-03-09 17:39:12 proc_lsst.multi:146 [INFO]  found job 16966493 in provider astro
proc_lsst.multi INFO: found job 16966493 in provider astro
2024-03-09 17:39:12 proc_lsst.multi:201 [INFO]  cancelling 16966493 on provider astro
proc_lsst.multi INFO: cancelling 16966493 on provider astro
2024-03-09 17:39:12 proc_lsst.multi:146 [INFO]  found job 16966478 in provider astro
proc_lsst.multi INFO: found job 16966478 in provider astro
2024-03-09 17:39:12 proc_lsst.multi:201 [INFO]  cancelling 16966478 on provider astro
proc_lsst.multi INFO: cancelling 16966478 on provider astro
2024-03-09 17:39:12 proc_lsst.multi:146 [INFO]  found job 16966518 in provider escience_a40
proc_lsst.multi INFO: found job 16966518 in provider escience_a40
2024-03-09 17:39:12 proc_lsst.multi:201 [INFO]  cancelling 16966518 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966518 on provider escience_a40
2024-03-09 17:39:12 proc_lsst.multi:146 [INFO]  found job 16966484 in provider astro
proc_lsst.multi INFO: found job 16966484 in provider astro
2024-03-09 17:39:12 proc_lsst.multi:201 [INFO]  cancelling 16966484 on provider astro
proc_lsst.multi INFO: cancelling 16966484 on provider astro
2024-03-09 17:39:12 proc_lsst.multi:146 [INFO]  found job 16966475 in provider astro
proc_lsst.multi INFO: found job 16966475 in provider astro
2024-03-09 17:39:12 proc_lsst.multi:201 [INFO]  cancelling 16966475 on provider astro
proc_lsst.multi INFO: cancelling 16966475 on provider astro
2024-03-09 17:39:13 proc_lsst.multi:146 [INFO]  found job 16966519 in provider escience_a40
proc_lsst.multi INFO: found job 16966519 in provider escience_a40
2024-03-09 17:39:13 proc_lsst.multi:201 [INFO]  cancelling 16966519 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966519 on provider escience_a40
2024-03-09 17:39:13 proc_lsst.multi:146 [INFO]  found job 16966500 in provider astro
proc_lsst.multi INFO: found job 16966500 in provider astro
2024-03-09 17:39:13 proc_lsst.multi:201 [INFO]  cancelling 16966500 on provider astro
proc_lsst.multi INFO: cancelling 16966500 on provider astro
2024-03-09 17:39:13 proc_lsst.multi:146 [INFO]  found job 16966481 in provider astro
proc_lsst.multi INFO: found job 16966481 in provider astro
2024-03-09 17:39:13 proc_lsst.multi:201 [INFO]  cancelling 16966481 on provider astro
proc_lsst.multi INFO: cancelling 16966481 on provider astro
2024-03-09 17:39:13 proc_lsst.multi:146 [INFO]  found job 16966489 in provider astro
proc_lsst.multi INFO: found job 16966489 in provider astro
2024-03-09 17:39:13 proc_lsst.multi:201 [INFO]  cancelling 16966489 on provider astro
proc_lsst.multi INFO: cancelling 16966489 on provider astro
2024-03-09 17:39:13 proc_lsst.multi:146 [INFO]  found job 16966507 in provider astro
proc_lsst.multi INFO: found job 16966507 in provider astro
2024-03-09 17:39:13 proc_lsst.multi:201 [INFO]  cancelling 16966507 on provider astro
proc_lsst.multi INFO: cancelling 16966507 on provider astro
2024-03-09 17:39:13 proc_lsst.multi:146 [INFO]  found job 16966487 in provider astro
proc_lsst.multi INFO: found job 16966487 in provider astro
2024-03-09 17:39:13 proc_lsst.multi:201 [INFO]  cancelling 16966487 on provider astro
proc_lsst.multi INFO: cancelling 16966487 on provider astro
2024-03-09 17:39:14 proc_lsst.multi:146 [INFO]  found job 16966520 in provider escience_a40
proc_lsst.multi INFO: found job 16966520 in provider escience_a40
2024-03-09 17:39:14 proc_lsst.multi:201 [INFO]  cancelling 16966520 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966520 on provider escience_a40
2024-03-09 17:39:14 proc_lsst.multi:146 [INFO]  found job 16966523 in provider escience_a40
proc_lsst.multi INFO: found job 16966523 in provider escience_a40
2024-03-09 17:39:14 proc_lsst.multi:201 [INFO]  cancelling 16966523 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966523 on provider escience_a40
2024-03-09 17:39:14 proc_lsst.multi:146 [INFO]  found job 16966492 in provider astro
proc_lsst.multi INFO: found job 16966492 in provider astro
2024-03-09 17:39:14 proc_lsst.multi:201 [INFO]  cancelling 16966492 on provider astro
proc_lsst.multi INFO: cancelling 16966492 on provider astro
2024-03-09 17:39:14 proc_lsst.multi:146 [INFO]  found job 16966509 in provider astro
proc_lsst.multi INFO: found job 16966509 in provider astro
2024-03-09 17:39:14 proc_lsst.multi:201 [INFO]  cancelling 16966509 on provider astro
proc_lsst.multi INFO: cancelling 16966509 on provider astro
2024-03-09 17:39:14 proc_lsst.multi:146 [INFO]  found job 16966525 in provider escience_a40
proc_lsst.multi INFO: found job 16966525 in provider escience_a40
2024-03-09 17:39:14 proc_lsst.multi:201 [INFO]  cancelling 16966525 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966525 on provider escience_a40
2024-03-09 17:39:15 proc_lsst.multi:146 [INFO]  found job 16966502 in provider astro
proc_lsst.multi INFO: found job 16966502 in provider astro
2024-03-09 17:39:15 proc_lsst.multi:201 [INFO]  cancelling 16966502 on provider astro
proc_lsst.multi INFO: cancelling 16966502 on provider astro
2024-03-09 17:39:15 proc_lsst.multi:146 [INFO]  found job 16966480 in provider astro
proc_lsst.multi INFO: found job 16966480 in provider astro
2024-03-09 17:39:15 proc_lsst.multi:201 [INFO]  cancelling 16966480 on provider astro
proc_lsst.multi INFO: cancelling 16966480 on provider astro
2024-03-09 17:39:15 proc_lsst.multi:146 [INFO]  found job 16966503 in provider astro
proc_lsst.multi INFO: found job 16966503 in provider astro
2024-03-09 17:39:15 proc_lsst.multi:201 [INFO]  cancelling 16966503 on provider astro
proc_lsst.multi INFO: cancelling 16966503 on provider astro
2024-03-09 17:39:15 proc_lsst.multi:146 [INFO]  found job 16966531 in provider escience_a40
proc_lsst.multi INFO: found job 16966531 in provider escience_a40
2024-03-09 17:39:15 proc_lsst.multi:201 [INFO]  cancelling 16966531 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966531 on provider escience_a40
2024-03-09 17:39:16 proc_lsst.multi:146 [INFO]  found job 16966528 in provider escience_a40
proc_lsst.multi INFO: found job 16966528 in provider escience_a40
2024-03-09 17:39:16 proc_lsst.multi:201 [INFO]  cancelling 16966528 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966528 on provider escience_a40
2024-03-09 17:39:16 proc_lsst.multi:146 [INFO]  found job 16966529 in provider escience_a40
proc_lsst.multi INFO: found job 16966529 in provider escience_a40
2024-03-09 17:39:16 proc_lsst.multi:201 [INFO]  cancelling 16966529 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966529 on provider escience_a40
2024-03-09 17:39:16 proc_lsst.multi:146 [INFO]  found job 16966499 in provider astro
proc_lsst.multi INFO: found job 16966499 in provider astro
2024-03-09 17:39:16 proc_lsst.multi:201 [INFO]  cancelling 16966499 on provider astro
proc_lsst.multi INFO: cancelling 16966499 on provider astro
2024-03-09 17:39:16 proc_lsst.multi:146 [INFO]  found job 16966533 in provider escience_a40
proc_lsst.multi INFO: found job 16966533 in provider escience_a40
2024-03-09 17:39:16 proc_lsst.multi:201 [INFO]  cancelling 16966533 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966533 on provider escience_a40
2024-03-09 17:39:16 proc_lsst.multi:146 [INFO]  found job 16966485 in provider astro
proc_lsst.multi INFO: found job 16966485 in provider astro
2024-03-09 17:39:16 proc_lsst.multi:201 [INFO]  cancelling 16966485 on provider astro
proc_lsst.multi INFO: cancelling 16966485 on provider astro
2024-03-09 17:39:17 proc_lsst.multi:146 [INFO]  found job 16966530 in provider escience_a40
proc_lsst.multi INFO: found job 16966530 in provider escience_a40
2024-03-09 17:39:17 proc_lsst.multi:201 [INFO]  cancelling 16966530 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966530 on provider escience_a40
2024-03-09 17:39:17 proc_lsst.multi:146 [INFO]  found job 16966532 in provider escience_a40
proc_lsst.multi INFO: found job 16966532 in provider escience_a40
2024-03-09 17:39:17 proc_lsst.multi:201 [INFO]  cancelling 16966532 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966532 on provider escience_a40
2024-03-09 17:39:17 proc_lsst.multi:146 [INFO]  found job 16966534 in provider escience_a40
proc_lsst.multi INFO: found job 16966534 in provider escience_a40
2024-03-09 17:39:17 proc_lsst.multi:201 [INFO]  cancelling 16966534 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966534 on provider escience_a40
2024-03-09 17:39:17 proc_lsst.multi:146 [INFO]  found job 16966527 in provider escience_a40
proc_lsst.multi INFO: found job 16966527 in provider escience_a40
2024-03-09 17:39:17 proc_lsst.multi:201 [INFO]  cancelling 16966527 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966527 on provider escience_a40
2024-03-09 17:39:17 proc_lsst.multi:146 [INFO]  found job 16966526 in provider escience_a40
proc_lsst.multi INFO: found job 16966526 in provider escience_a40
2024-03-09 17:39:17 proc_lsst.multi:201 [INFO]  cancelling 16966526 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966526 on provider escience_a40
2024-03-09 17:39:18 proc_lsst.multi:146 [INFO]  found job 16966512 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966512 in provider escience_rtx6k
2024-03-09 17:39:18 proc_lsst.multi:201 [INFO]  cancelling 16966512 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16966512 on provider escience_rtx6k
2024-03-09 17:39:18 proc_lsst.multi:146 [INFO]  found job 16966511 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966511 in provider escience_rtx6k
2024-03-09 17:39:18 proc_lsst.multi:201 [INFO]  cancelling 16966511 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16966511 on provider escience_rtx6k
2024-03-09 17:39:18 proc_lsst.multi:146 [INFO]  found job 64340 in provider local
proc_lsst.multi INFO: found job 64340 in provider local
2024-03-09 17:39:18 proc_lsst.multi:201 [INFO]  cancelling 64340 on provider local
proc_lsst.multi INFO: cancelling 64340 on provider local
parsl.providers.local.local DEBUG: Terminating job/proc_id: 64340
2024-03-09 17:39:18 proc_lsst.multi:146 [INFO]  found job 16966476 in provider astro
proc_lsst.multi INFO: found job 16966476 in provider astro
2024-03-09 17:39:18 proc_lsst.multi:201 [INFO]  cancelling 16966476 on provider astro
proc_lsst.multi INFO: cancelling 16966476 on provider astro
2024-03-09 17:39:18 proc_lsst.multi:146 [INFO]  found job 16966477 in provider astro
proc_lsst.multi INFO: found job 16966477 in provider astro
2024-03-09 17:39:18 proc_lsst.multi:201 [INFO]  cancelling 16966477 on provider astro
proc_lsst.multi INFO: cancelling 16966477 on provider astro
2024-03-09 17:39:18 proc_lsst.multi:146 [INFO]  found job 16966554 in provider checkpoint
proc_lsst.multi INFO: found job 16966554 in provider checkpoint
2024-03-09 17:39:18 proc_lsst.multi:201 [INFO]  cancelling 16966554 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966554 on provider checkpoint
2024-03-09 17:39:19 proc_lsst.multi:146 [INFO]  found job 16966553 in provider checkpoint
proc_lsst.multi INFO: found job 16966553 in provider checkpoint
2024-03-09 17:39:19 proc_lsst.multi:201 [INFO]  cancelling 16966553 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966553 on provider checkpoint
2024-03-09 17:39:19 proc_lsst.multi:146 [INFO]  found job 16966555 in provider checkpoint
proc_lsst.multi INFO: found job 16966555 in provider checkpoint
2024-03-09 17:39:19 proc_lsst.multi:201 [INFO]  cancelling 16966555 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966555 on provider checkpoint
2024-03-09 17:39:19 proc_lsst.multi:146 [INFO]  found job 16966557 in provider checkpoint
proc_lsst.multi INFO: found job 16966557 in provider checkpoint
2024-03-09 17:39:19 proc_lsst.multi:201 [INFO]  cancelling 16966557 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966557 on provider checkpoint
2024-03-09 17:39:19 proc_lsst.multi:146 [INFO]  found job 16966558 in provider checkpoint
proc_lsst.multi INFO: found job 16966558 in provider checkpoint
2024-03-09 17:39:19 proc_lsst.multi:201 [INFO]  cancelling 16966558 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966558 on provider checkpoint
2024-03-09 17:39:19 proc_lsst.multi:146 [INFO]  found job 16966559 in provider checkpoint
proc_lsst.multi INFO: found job 16966559 in provider checkpoint
2024-03-09 17:39:19 proc_lsst.multi:201 [INFO]  cancelling 16966559 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966559 on provider checkpoint
2024-03-09 17:39:19 proc_lsst.multi:146 [INFO]  found job 16966560 in provider checkpoint
proc_lsst.multi INFO: found job 16966560 in provider checkpoint
2024-03-09 17:39:19 proc_lsst.multi:201 [INFO]  cancelling 16966560 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966560 on provider checkpoint
2024-03-09 17:39:20 proc_lsst.multi:146 [INFO]  found job 16966556 in provider checkpoint
proc_lsst.multi INFO: found job 16966556 in provider checkpoint
2024-03-09 17:39:20 proc_lsst.multi:201 [INFO]  cancelling 16966556 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966556 on provider checkpoint
2024-03-09 17:39:20 proc_lsst.multi:146 [INFO]  found job 16966551 in provider checkpoint
proc_lsst.multi INFO: found job 16966551 in provider checkpoint
2024-03-09 17:39:20 proc_lsst.multi:201 [INFO]  cancelling 16966551 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966551 on provider checkpoint
2024-03-09 17:39:20 proc_lsst.multi:146 [INFO]  found job 16966550 in provider checkpoint
proc_lsst.multi INFO: found job 16966550 in provider checkpoint
2024-03-09 17:39:20 proc_lsst.multi:201 [INFO]  cancelling 16966550 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966550 on provider checkpoint
2024-03-09 17:39:20 proc_lsst.multi:146 [INFO]  found job 16966537 in provider checkpoint
proc_lsst.multi INFO: found job 16966537 in provider checkpoint
2024-03-09 17:39:20 proc_lsst.multi:201 [INFO]  cancelling 16966537 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966537 on provider checkpoint
2024-03-09 17:39:20 proc_lsst.multi:146 [INFO]  found job 16966544 in provider checkpoint
proc_lsst.multi INFO: found job 16966544 in provider checkpoint
2024-03-09 17:39:20 proc_lsst.multi:201 [INFO]  cancelling 16966544 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966544 on provider checkpoint
2024-03-09 17:39:21 proc_lsst.multi:146 [INFO]  found job 16966542 in provider checkpoint
proc_lsst.multi INFO: found job 16966542 in provider checkpoint
2024-03-09 17:39:21 proc_lsst.multi:201 [INFO]  cancelling 16966542 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966542 on provider checkpoint
2024-03-09 17:39:21 proc_lsst.multi:146 [INFO]  found job 16966546 in provider checkpoint
proc_lsst.multi INFO: found job 16966546 in provider checkpoint
2024-03-09 17:39:21 proc_lsst.multi:201 [INFO]  cancelling 16966546 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966546 on provider checkpoint
2024-03-09 17:39:21 proc_lsst.multi:146 [INFO]  found job 16966543 in provider checkpoint
proc_lsst.multi INFO: found job 16966543 in provider checkpoint
2024-03-09 17:39:21 proc_lsst.multi:201 [INFO]  cancelling 16966543 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966543 on provider checkpoint
2024-03-09 17:39:21 proc_lsst.multi:146 [INFO]  found job 16966547 in provider checkpoint
proc_lsst.multi INFO: found job 16966547 in provider checkpoint
2024-03-09 17:39:21 proc_lsst.multi:201 [INFO]  cancelling 16966547 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966547 on provider checkpoint
2024-03-09 17:39:21 proc_lsst.multi:146 [INFO]  found job 16966517 in provider escience_a40
proc_lsst.multi INFO: found job 16966517 in provider escience_a40
2024-03-09 17:39:21 proc_lsst.multi:201 [INFO]  cancelling 16966517 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966517 on provider escience_a40
parsl.dataflow.dflow INFO: Shutting down executor multi
2024-03-09 17:39:21 proc_lsst.multi:40 [INFO]  Cancelling all provider resources
proc_lsst.multi INFO: Cancelling all provider resources
2024-03-09 17:39:21 proc_lsst.multi:47 [INFO]  new jobs since last cancel ['16966535', '16966536', '16966537', '16966538', '16966539', '16966540', '16966541', '16966542', '16966543', '16966544', '16966545', '16966546', '16966547', '16966548', '16966549', '16966550', '16966551', '16966553', '16966554', '16966555', '16966556', '16966557', '16966558', '16966559', '16966560', '16966561', '16966562', '16966563', '16966564', '16966565', '16966566', '16966567', '16966568', '16966569', '16966570', '16966571', '16966572', '16966475', '16966476', '16966477', '16966478', '16966479', '16966480', '16966481', '16966482', '16966483', '16966484', '16966485', '16966486', '16966487', '16966488', '16966489', '16966490', '16966491', '16966492', '16966493', '16966499', '16966500', '16966501', '16966502', '16966503', '16966504', '16966505', '16966506', '16966507', '16966508', '16966509', '16966515', '16966516', '16966517', '16966518', '16966519', '16966520', '16966521', '16966522', '16966523', '16966524', '16966525', '16966526', '16966527', '16966528', '16966529', '16966530', '16966531', '16966532', '16966533', '16966534', '16966510', '16966511', '16966512', '16966513', '16966514', '64340']
proc_lsst.multi INFO: new jobs since last cancel ['16966535', '16966536', '16966537', '16966538', '16966539', '16966540', '16966541', '16966542', '16966543', '16966544', '16966545', '16966546', '16966547', '16966548', '16966549', '16966550', '16966551', '16966553', '16966554', '16966555', '16966556', '16966557', '16966558', '16966559', '16966560', '16966561', '16966562', '16966563', '16966564', '16966565', '16966566', '16966567', '16966568', '16966569', '16966570', '16966571', '16966572', '16966475', '16966476', '16966477', '16966478', '16966479', '16966480', '16966481', '16966482', '16966483', '16966484', '16966485', '16966486', '16966487', '16966488', '16966489', '16966490', '16966491', '16966492', '16966493', '16966499', '16966500', '16966501', '16966502', '16966503', '16966504', '16966505', '16966506', '16966507', '16966508', '16966509', '16966515', '16966516', '16966517', '16966518', '16966519', '16966520', '16966521', '16966522', '16966523', '16966524', '16966525', '16966526', '16966527', '16966528', '16966529', '16966530', '16966531', '16966532', '16966533', '16966534', '16966510', '16966511', '16966512', '16966513', '16966514', '64340']
2024-03-09 17:39:21 proc_lsst.multi:146 [INFO]  found job 16966535 in provider checkpoint
proc_lsst.multi INFO: found job 16966535 in provider checkpoint
2024-03-09 17:39:21 proc_lsst.multi:201 [INFO]  cancelling 16966535 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966535 on provider checkpoint
2024-03-09 17:39:22 proc_lsst.multi:146 [INFO]  found job 16966536 in provider checkpoint
proc_lsst.multi INFO: found job 16966536 in provider checkpoint
2024-03-09 17:39:22 proc_lsst.multi:201 [INFO]  cancelling 16966536 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966536 on provider checkpoint
2024-03-09 17:39:22 proc_lsst.multi:146 [INFO]  found job 16966537 in provider checkpoint
proc_lsst.multi INFO: found job 16966537 in provider checkpoint
2024-03-09 17:39:22 proc_lsst.multi:201 [INFO]  cancelling 16966537 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966537 on provider checkpoint
2024-03-09 17:39:22 proc_lsst.multi:146 [INFO]  found job 16966538 in provider checkpoint
proc_lsst.multi INFO: found job 16966538 in provider checkpoint
2024-03-09 17:39:22 proc_lsst.multi:201 [INFO]  cancelling 16966538 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966538 on provider checkpoint
2024-03-09 17:39:22 proc_lsst.multi:146 [INFO]  found job 16966539 in provider checkpoint
proc_lsst.multi INFO: found job 16966539 in provider checkpoint
2024-03-09 17:39:22 proc_lsst.multi:201 [INFO]  cancelling 16966539 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966539 on provider checkpoint
2024-03-09 17:39:22 proc_lsst.multi:146 [INFO]  found job 16966540 in provider checkpoint
proc_lsst.multi INFO: found job 16966540 in provider checkpoint
2024-03-09 17:39:22 proc_lsst.multi:201 [INFO]  cancelling 16966540 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966540 on provider checkpoint
2024-03-09 17:39:22 proc_lsst.multi:146 [INFO]  found job 16966541 in provider checkpoint
proc_lsst.multi INFO: found job 16966541 in provider checkpoint
2024-03-09 17:39:22 proc_lsst.multi:201 [INFO]  cancelling 16966541 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966541 on provider checkpoint
2024-03-09 17:39:23 proc_lsst.multi:146 [INFO]  found job 16966542 in provider checkpoint
proc_lsst.multi INFO: found job 16966542 in provider checkpoint
2024-03-09 17:39:23 proc_lsst.multi:201 [INFO]  cancelling 16966542 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966542 on provider checkpoint
2024-03-09 17:39:23 proc_lsst.multi:146 [INFO]  found job 16966543 in provider checkpoint
proc_lsst.multi INFO: found job 16966543 in provider checkpoint
2024-03-09 17:39:23 proc_lsst.multi:201 [INFO]  cancelling 16966543 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966543 on provider checkpoint
2024-03-09 17:39:23 proc_lsst.multi:146 [INFO]  found job 16966544 in provider checkpoint
proc_lsst.multi INFO: found job 16966544 in provider checkpoint
2024-03-09 17:39:23 proc_lsst.multi:201 [INFO]  cancelling 16966544 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966544 on provider checkpoint
2024-03-09 17:39:23 proc_lsst.multi:146 [INFO]  found job 16966545 in provider checkpoint
proc_lsst.multi INFO: found job 16966545 in provider checkpoint
2024-03-09 17:39:23 proc_lsst.multi:201 [INFO]  cancelling 16966545 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966545 on provider checkpoint
2024-03-09 17:39:23 proc_lsst.multi:146 [INFO]  found job 16966546 in provider checkpoint
proc_lsst.multi INFO: found job 16966546 in provider checkpoint
2024-03-09 17:39:23 proc_lsst.multi:201 [INFO]  cancelling 16966546 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966546 on provider checkpoint
2024-03-09 17:39:23 proc_lsst.multi:146 [INFO]  found job 16966547 in provider checkpoint
proc_lsst.multi INFO: found job 16966547 in provider checkpoint
2024-03-09 17:39:23 proc_lsst.multi:201 [INFO]  cancelling 16966547 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966547 on provider checkpoint
2024-03-09 17:39:24 proc_lsst.multi:146 [INFO]  found job 16966548 in provider checkpoint
proc_lsst.multi INFO: found job 16966548 in provider checkpoint
2024-03-09 17:39:24 proc_lsst.multi:201 [INFO]  cancelling 16966548 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966548 on provider checkpoint
2024-03-09 17:39:24 proc_lsst.multi:146 [INFO]  found job 16966549 in provider checkpoint
proc_lsst.multi INFO: found job 16966549 in provider checkpoint
2024-03-09 17:39:24 proc_lsst.multi:201 [INFO]  cancelling 16966549 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966549 on provider checkpoint
2024-03-09 17:39:24 proc_lsst.multi:146 [INFO]  found job 16966550 in provider checkpoint
proc_lsst.multi INFO: found job 16966550 in provider checkpoint
2024-03-09 17:39:24 proc_lsst.multi:201 [INFO]  cancelling 16966550 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966550 on provider checkpoint
2024-03-09 17:39:24 proc_lsst.multi:146 [INFO]  found job 16966551 in provider checkpoint
proc_lsst.multi INFO: found job 16966551 in provider checkpoint
2024-03-09 17:39:24 proc_lsst.multi:201 [INFO]  cancelling 16966551 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966551 on provider checkpoint
2024-03-09 17:39:24 proc_lsst.multi:146 [INFO]  found job 16966553 in provider checkpoint
proc_lsst.multi INFO: found job 16966553 in provider checkpoint
2024-03-09 17:39:24 proc_lsst.multi:201 [INFO]  cancelling 16966553 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966553 on provider checkpoint
2024-03-09 17:39:24 proc_lsst.multi:146 [INFO]  found job 16966554 in provider checkpoint
proc_lsst.multi INFO: found job 16966554 in provider checkpoint
2024-03-09 17:39:24 proc_lsst.multi:201 [INFO]  cancelling 16966554 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966554 on provider checkpoint
2024-03-09 17:39:25 proc_lsst.multi:146 [INFO]  found job 16966555 in provider checkpoint
proc_lsst.multi INFO: found job 16966555 in provider checkpoint
2024-03-09 17:39:25 proc_lsst.multi:201 [INFO]  cancelling 16966555 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966555 on provider checkpoint
2024-03-09 17:39:25 proc_lsst.multi:146 [INFO]  found job 16966556 in provider checkpoint
proc_lsst.multi INFO: found job 16966556 in provider checkpoint
2024-03-09 17:39:25 proc_lsst.multi:201 [INFO]  cancelling 16966556 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966556 on provider checkpoint
2024-03-09 17:39:25 proc_lsst.multi:146 [INFO]  found job 16966557 in provider checkpoint
proc_lsst.multi INFO: found job 16966557 in provider checkpoint
2024-03-09 17:39:25 proc_lsst.multi:201 [INFO]  cancelling 16966557 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966557 on provider checkpoint
2024-03-09 17:39:25 proc_lsst.multi:146 [INFO]  found job 16966558 in provider checkpoint
proc_lsst.multi INFO: found job 16966558 in provider checkpoint
2024-03-09 17:39:25 proc_lsst.multi:201 [INFO]  cancelling 16966558 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966558 on provider checkpoint
2024-03-09 17:39:25 proc_lsst.multi:146 [INFO]  found job 16966559 in provider checkpoint
proc_lsst.multi INFO: found job 16966559 in provider checkpoint
2024-03-09 17:39:25 proc_lsst.multi:201 [INFO]  cancelling 16966559 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966559 on provider checkpoint
2024-03-09 17:39:25 proc_lsst.multi:146 [INFO]  found job 16966560 in provider checkpoint
proc_lsst.multi INFO: found job 16966560 in provider checkpoint
2024-03-09 17:39:25 proc_lsst.multi:201 [INFO]  cancelling 16966560 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966560 on provider checkpoint
2024-03-09 17:39:26 proc_lsst.multi:146 [INFO]  found job 16966561 in provider checkpoint
proc_lsst.multi INFO: found job 16966561 in provider checkpoint
2024-03-09 17:39:26 proc_lsst.multi:201 [INFO]  cancelling 16966561 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966561 on provider checkpoint
2024-03-09 17:39:26 proc_lsst.multi:146 [INFO]  found job 16966562 in provider checkpoint
proc_lsst.multi INFO: found job 16966562 in provider checkpoint
2024-03-09 17:39:26 proc_lsst.multi:201 [INFO]  cancelling 16966562 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966562 on provider checkpoint
2024-03-09 17:39:26 proc_lsst.multi:146 [INFO]  found job 16966563 in provider checkpoint
proc_lsst.multi INFO: found job 16966563 in provider checkpoint
2024-03-09 17:39:26 proc_lsst.multi:201 [INFO]  cancelling 16966563 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966563 on provider checkpoint
2024-03-09 17:39:26 proc_lsst.multi:146 [INFO]  found job 16966564 in provider checkpoint
proc_lsst.multi INFO: found job 16966564 in provider checkpoint
2024-03-09 17:39:26 proc_lsst.multi:201 [INFO]  cancelling 16966564 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966564 on provider checkpoint
2024-03-09 17:39:26 proc_lsst.multi:146 [INFO]  found job 16966565 in provider checkpoint
proc_lsst.multi INFO: found job 16966565 in provider checkpoint
2024-03-09 17:39:26 proc_lsst.multi:201 [INFO]  cancelling 16966565 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966565 on provider checkpoint
2024-03-09 17:39:26 proc_lsst.multi:146 [INFO]  found job 16966566 in provider checkpoint
proc_lsst.multi INFO: found job 16966566 in provider checkpoint
2024-03-09 17:39:26 proc_lsst.multi:201 [INFO]  cancelling 16966566 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966566 on provider checkpoint
2024-03-09 17:39:26 proc_lsst.multi:146 [INFO]  found job 16966567 in provider checkpoint
proc_lsst.multi INFO: found job 16966567 in provider checkpoint
2024-03-09 17:39:26 proc_lsst.multi:201 [INFO]  cancelling 16966567 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966567 on provider checkpoint
2024-03-09 17:39:27 proc_lsst.multi:146 [INFO]  found job 16966568 in provider checkpoint
proc_lsst.multi INFO: found job 16966568 in provider checkpoint
2024-03-09 17:39:27 proc_lsst.multi:201 [INFO]  cancelling 16966568 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966568 on provider checkpoint
2024-03-09 17:39:27 proc_lsst.multi:146 [INFO]  found job 16966569 in provider checkpoint
proc_lsst.multi INFO: found job 16966569 in provider checkpoint
2024-03-09 17:39:27 proc_lsst.multi:201 [INFO]  cancelling 16966569 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966569 on provider checkpoint
2024-03-09 17:39:27 proc_lsst.multi:146 [INFO]  found job 16966570 in provider checkpoint
proc_lsst.multi INFO: found job 16966570 in provider checkpoint
2024-03-09 17:39:27 proc_lsst.multi:201 [INFO]  cancelling 16966570 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966570 on provider checkpoint
2024-03-09 17:39:29 proc_lsst.multi:146 [INFO]  found job 16966571 in provider checkpoint
proc_lsst.multi INFO: found job 16966571 in provider checkpoint
2024-03-09 17:39:29 proc_lsst.multi:201 [INFO]  cancelling 16966571 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966571 on provider checkpoint
2024-03-09 17:39:29 proc_lsst.multi:146 [INFO]  found job 16966572 in provider checkpoint
proc_lsst.multi INFO: found job 16966572 in provider checkpoint
2024-03-09 17:39:29 proc_lsst.multi:201 [INFO]  cancelling 16966572 on provider checkpoint
proc_lsst.multi INFO: cancelling 16966572 on provider checkpoint
2024-03-09 17:39:29 proc_lsst.multi:146 [INFO]  found job 16966475 in provider astro
proc_lsst.multi INFO: found job 16966475 in provider astro
2024-03-09 17:39:29 proc_lsst.multi:201 [INFO]  cancelling 16966475 on provider astro
proc_lsst.multi INFO: cancelling 16966475 on provider astro
2024-03-09 17:39:30 proc_lsst.multi:146 [INFO]  found job 16966476 in provider astro
proc_lsst.multi INFO: found job 16966476 in provider astro
2024-03-09 17:39:30 proc_lsst.multi:201 [INFO]  cancelling 16966476 on provider astro
proc_lsst.multi INFO: cancelling 16966476 on provider astro
2024-03-09 17:39:30 proc_lsst.multi:146 [INFO]  found job 16966477 in provider astro
proc_lsst.multi INFO: found job 16966477 in provider astro
2024-03-09 17:39:30 proc_lsst.multi:201 [INFO]  cancelling 16966477 on provider astro
proc_lsst.multi INFO: cancelling 16966477 on provider astro
2024-03-09 17:39:30 proc_lsst.multi:146 [INFO]  found job 16966478 in provider astro
proc_lsst.multi INFO: found job 16966478 in provider astro
2024-03-09 17:39:30 proc_lsst.multi:201 [INFO]  cancelling 16966478 on provider astro
proc_lsst.multi INFO: cancelling 16966478 on provider astro
2024-03-09 17:39:30 proc_lsst.multi:146 [INFO]  found job 16966479 in provider astro
proc_lsst.multi INFO: found job 16966479 in provider astro
2024-03-09 17:39:30 proc_lsst.multi:201 [INFO]  cancelling 16966479 on provider astro
proc_lsst.multi INFO: cancelling 16966479 on provider astro
2024-03-09 17:39:30 proc_lsst.multi:146 [INFO]  found job 16966480 in provider astro
proc_lsst.multi INFO: found job 16966480 in provider astro
2024-03-09 17:39:30 proc_lsst.multi:201 [INFO]  cancelling 16966480 on provider astro
proc_lsst.multi INFO: cancelling 16966480 on provider astro
2024-03-09 17:39:31 proc_lsst.multi:146 [INFO]  found job 16966481 in provider astro
proc_lsst.multi INFO: found job 16966481 in provider astro
2024-03-09 17:39:31 proc_lsst.multi:201 [INFO]  cancelling 16966481 on provider astro
proc_lsst.multi INFO: cancelling 16966481 on provider astro
2024-03-09 17:39:31 proc_lsst.multi:146 [INFO]  found job 16966482 in provider astro
proc_lsst.multi INFO: found job 16966482 in provider astro
2024-03-09 17:39:31 proc_lsst.multi:201 [INFO]  cancelling 16966482 on provider astro
proc_lsst.multi INFO: cancelling 16966482 on provider astro
2024-03-09 17:39:31 proc_lsst.multi:146 [INFO]  found job 16966483 in provider astro
proc_lsst.multi INFO: found job 16966483 in provider astro
2024-03-09 17:39:31 proc_lsst.multi:201 [INFO]  cancelling 16966483 on provider astro
proc_lsst.multi INFO: cancelling 16966483 on provider astro
2024-03-09 17:39:31 proc_lsst.multi:146 [INFO]  found job 16966484 in provider astro
proc_lsst.multi INFO: found job 16966484 in provider astro
2024-03-09 17:39:31 proc_lsst.multi:201 [INFO]  cancelling 16966484 on provider astro
proc_lsst.multi INFO: cancelling 16966484 on provider astro
2024-03-09 17:39:31 proc_lsst.multi:146 [INFO]  found job 16966485 in provider astro
proc_lsst.multi INFO: found job 16966485 in provider astro
2024-03-09 17:39:31 proc_lsst.multi:201 [INFO]  cancelling 16966485 on provider astro
proc_lsst.multi INFO: cancelling 16966485 on provider astro
2024-03-09 17:39:31 proc_lsst.multi:146 [INFO]  found job 16966486 in provider astro
proc_lsst.multi INFO: found job 16966486 in provider astro
2024-03-09 17:39:31 proc_lsst.multi:201 [INFO]  cancelling 16966486 on provider astro
proc_lsst.multi INFO: cancelling 16966486 on provider astro
2024-03-09 17:39:32 proc_lsst.multi:146 [INFO]  found job 16966487 in provider astro
proc_lsst.multi INFO: found job 16966487 in provider astro
2024-03-09 17:39:32 proc_lsst.multi:201 [INFO]  cancelling 16966487 on provider astro
proc_lsst.multi INFO: cancelling 16966487 on provider astro
2024-03-09 17:39:32 proc_lsst.multi:146 [INFO]  found job 16966488 in provider astro
proc_lsst.multi INFO: found job 16966488 in provider astro
2024-03-09 17:39:32 proc_lsst.multi:201 [INFO]  cancelling 16966488 on provider astro
proc_lsst.multi INFO: cancelling 16966488 on provider astro
2024-03-09 17:39:32 proc_lsst.multi:146 [INFO]  found job 16966489 in provider astro
proc_lsst.multi INFO: found job 16966489 in provider astro
2024-03-09 17:39:32 proc_lsst.multi:201 [INFO]  cancelling 16966489 on provider astro
proc_lsst.multi INFO: cancelling 16966489 on provider astro
2024-03-09 17:39:32 proc_lsst.multi:146 [INFO]  found job 16966490 in provider astro
proc_lsst.multi INFO: found job 16966490 in provider astro
2024-03-09 17:39:32 proc_lsst.multi:201 [INFO]  cancelling 16966490 on provider astro
proc_lsst.multi INFO: cancelling 16966490 on provider astro
2024-03-09 17:39:32 proc_lsst.multi:146 [INFO]  found job 16966491 in provider astro
proc_lsst.multi INFO: found job 16966491 in provider astro
2024-03-09 17:39:32 proc_lsst.multi:201 [INFO]  cancelling 16966491 on provider astro
proc_lsst.multi INFO: cancelling 16966491 on provider astro
2024-03-09 17:39:32 proc_lsst.multi:146 [INFO]  found job 16966492 in provider astro
proc_lsst.multi INFO: found job 16966492 in provider astro
2024-03-09 17:39:32 proc_lsst.multi:201 [INFO]  cancelling 16966492 on provider astro
proc_lsst.multi INFO: cancelling 16966492 on provider astro
2024-03-09 17:39:33 proc_lsst.multi:146 [INFO]  found job 16966493 in provider astro
proc_lsst.multi INFO: found job 16966493 in provider astro
2024-03-09 17:39:33 proc_lsst.multi:201 [INFO]  cancelling 16966493 on provider astro
proc_lsst.multi INFO: cancelling 16966493 on provider astro
2024-03-09 17:39:33 proc_lsst.multi:146 [INFO]  found job 16966499 in provider astro
proc_lsst.multi INFO: found job 16966499 in provider astro
2024-03-09 17:39:33 proc_lsst.multi:201 [INFO]  cancelling 16966499 on provider astro
proc_lsst.multi INFO: cancelling 16966499 on provider astro
2024-03-09 17:39:33 proc_lsst.multi:146 [INFO]  found job 16966500 in provider astro
proc_lsst.multi INFO: found job 16966500 in provider astro
2024-03-09 17:39:33 proc_lsst.multi:201 [INFO]  cancelling 16966500 on provider astro
proc_lsst.multi INFO: cancelling 16966500 on provider astro
2024-03-09 17:39:33 proc_lsst.multi:146 [INFO]  found job 16966501 in provider astro
proc_lsst.multi INFO: found job 16966501 in provider astro
2024-03-09 17:39:33 proc_lsst.multi:201 [INFO]  cancelling 16966501 on provider astro
proc_lsst.multi INFO: cancelling 16966501 on provider astro
2024-03-09 17:39:33 proc_lsst.multi:146 [INFO]  found job 16966502 in provider astro
proc_lsst.multi INFO: found job 16966502 in provider astro
2024-03-09 17:39:33 proc_lsst.multi:201 [INFO]  cancelling 16966502 on provider astro
proc_lsst.multi INFO: cancelling 16966502 on provider astro
2024-03-09 17:39:33 proc_lsst.multi:146 [INFO]  found job 16966503 in provider astro
proc_lsst.multi INFO: found job 16966503 in provider astro
2024-03-09 17:39:33 proc_lsst.multi:201 [INFO]  cancelling 16966503 on provider astro
proc_lsst.multi INFO: cancelling 16966503 on provider astro
2024-03-09 17:39:34 proc_lsst.multi:146 [INFO]  found job 16966504 in provider astro
proc_lsst.multi INFO: found job 16966504 in provider astro
2024-03-09 17:39:34 proc_lsst.multi:201 [INFO]  cancelling 16966504 on provider astro
proc_lsst.multi INFO: cancelling 16966504 on provider astro
2024-03-09 17:39:34 proc_lsst.multi:146 [INFO]  found job 16966505 in provider astro
proc_lsst.multi INFO: found job 16966505 in provider astro
2024-03-09 17:39:34 proc_lsst.multi:201 [INFO]  cancelling 16966505 on provider astro
proc_lsst.multi INFO: cancelling 16966505 on provider astro
2024-03-09 17:39:34 proc_lsst.multi:146 [INFO]  found job 16966506 in provider astro
proc_lsst.multi INFO: found job 16966506 in provider astro
2024-03-09 17:39:34 proc_lsst.multi:201 [INFO]  cancelling 16966506 on provider astro
proc_lsst.multi INFO: cancelling 16966506 on provider astro
2024-03-09 17:39:34 proc_lsst.multi:146 [INFO]  found job 16966507 in provider astro
proc_lsst.multi INFO: found job 16966507 in provider astro
2024-03-09 17:39:34 proc_lsst.multi:201 [INFO]  cancelling 16966507 on provider astro
proc_lsst.multi INFO: cancelling 16966507 on provider astro
2024-03-09 17:39:34 proc_lsst.multi:146 [INFO]  found job 16966508 in provider astro
proc_lsst.multi INFO: found job 16966508 in provider astro
2024-03-09 17:39:34 proc_lsst.multi:201 [INFO]  cancelling 16966508 on provider astro
proc_lsst.multi INFO: cancelling 16966508 on provider astro
2024-03-09 17:39:34 proc_lsst.multi:146 [INFO]  found job 16966509 in provider astro
proc_lsst.multi INFO: found job 16966509 in provider astro
2024-03-09 17:39:34 proc_lsst.multi:201 [INFO]  cancelling 16966509 on provider astro
proc_lsst.multi INFO: cancelling 16966509 on provider astro
2024-03-09 17:39:34 proc_lsst.multi:146 [INFO]  found job 16966515 in provider escience_a40
proc_lsst.multi INFO: found job 16966515 in provider escience_a40
2024-03-09 17:39:34 proc_lsst.multi:201 [INFO]  cancelling 16966515 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966515 on provider escience_a40
2024-03-09 17:39:35 proc_lsst.multi:146 [INFO]  found job 16966516 in provider escience_a40
proc_lsst.multi INFO: found job 16966516 in provider escience_a40
2024-03-09 17:39:35 proc_lsst.multi:201 [INFO]  cancelling 16966516 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966516 on provider escience_a40
2024-03-09 17:39:35 proc_lsst.multi:146 [INFO]  found job 16966517 in provider escience_a40
proc_lsst.multi INFO: found job 16966517 in provider escience_a40
2024-03-09 17:39:35 proc_lsst.multi:201 [INFO]  cancelling 16966517 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966517 on provider escience_a40
2024-03-09 17:39:35 proc_lsst.multi:146 [INFO]  found job 16966518 in provider escience_a40
proc_lsst.multi INFO: found job 16966518 in provider escience_a40
2024-03-09 17:39:35 proc_lsst.multi:201 [INFO]  cancelling 16966518 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966518 on provider escience_a40
2024-03-09 17:39:35 proc_lsst.multi:146 [INFO]  found job 16966519 in provider escience_a40
proc_lsst.multi INFO: found job 16966519 in provider escience_a40
2024-03-09 17:39:35 proc_lsst.multi:201 [INFO]  cancelling 16966519 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966519 on provider escience_a40
2024-03-09 17:39:35 proc_lsst.multi:146 [INFO]  found job 16966520 in provider escience_a40
proc_lsst.multi INFO: found job 16966520 in provider escience_a40
2024-03-09 17:39:35 proc_lsst.multi:201 [INFO]  cancelling 16966520 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966520 on provider escience_a40
2024-03-09 17:39:35 proc_lsst.multi:146 [INFO]  found job 16966521 in provider escience_a40
proc_lsst.multi INFO: found job 16966521 in provider escience_a40
2024-03-09 17:39:35 proc_lsst.multi:201 [INFO]  cancelling 16966521 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966521 on provider escience_a40
2024-03-09 17:39:36 proc_lsst.multi:146 [INFO]  found job 16966522 in provider escience_a40
proc_lsst.multi INFO: found job 16966522 in provider escience_a40
2024-03-09 17:39:36 proc_lsst.multi:201 [INFO]  cancelling 16966522 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966522 on provider escience_a40
2024-03-09 17:39:36 proc_lsst.multi:146 [INFO]  found job 16966523 in provider escience_a40
proc_lsst.multi INFO: found job 16966523 in provider escience_a40
2024-03-09 17:39:36 proc_lsst.multi:201 [INFO]  cancelling 16966523 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966523 on provider escience_a40
2024-03-09 17:39:36 proc_lsst.multi:146 [INFO]  found job 16966524 in provider escience_a40
proc_lsst.multi INFO: found job 16966524 in provider escience_a40
2024-03-09 17:39:36 proc_lsst.multi:201 [INFO]  cancelling 16966524 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966524 on provider escience_a40
2024-03-09 17:39:36 proc_lsst.multi:146 [INFO]  found job 16966525 in provider escience_a40
proc_lsst.multi INFO: found job 16966525 in provider escience_a40
2024-03-09 17:39:36 proc_lsst.multi:201 [INFO]  cancelling 16966525 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966525 on provider escience_a40
2024-03-09 17:39:36 proc_lsst.multi:146 [INFO]  found job 16966526 in provider escience_a40
proc_lsst.multi INFO: found job 16966526 in provider escience_a40
2024-03-09 17:39:36 proc_lsst.multi:201 [INFO]  cancelling 16966526 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966526 on provider escience_a40
2024-03-09 17:39:36 proc_lsst.multi:146 [INFO]  found job 16966527 in provider escience_a40
proc_lsst.multi INFO: found job 16966527 in provider escience_a40
2024-03-09 17:39:36 proc_lsst.multi:201 [INFO]  cancelling 16966527 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966527 on provider escience_a40
2024-03-09 17:39:37 proc_lsst.multi:146 [INFO]  found job 16966528 in provider escience_a40
proc_lsst.multi INFO: found job 16966528 in provider escience_a40
2024-03-09 17:39:37 proc_lsst.multi:201 [INFO]  cancelling 16966528 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966528 on provider escience_a40
2024-03-09 17:39:37 proc_lsst.multi:146 [INFO]  found job 16966529 in provider escience_a40
proc_lsst.multi INFO: found job 16966529 in provider escience_a40
2024-03-09 17:39:37 proc_lsst.multi:201 [INFO]  cancelling 16966529 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966529 on provider escience_a40
2024-03-09 17:39:37 proc_lsst.multi:146 [INFO]  found job 16966530 in provider escience_a40
proc_lsst.multi INFO: found job 16966530 in provider escience_a40
2024-03-09 17:39:37 proc_lsst.multi:201 [INFO]  cancelling 16966530 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966530 on provider escience_a40
2024-03-09 17:39:37 proc_lsst.multi:146 [INFO]  found job 16966531 in provider escience_a40
proc_lsst.multi INFO: found job 16966531 in provider escience_a40
2024-03-09 17:39:37 proc_lsst.multi:201 [INFO]  cancelling 16966531 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966531 on provider escience_a40
2024-03-09 17:39:37 proc_lsst.multi:146 [INFO]  found job 16966532 in provider escience_a40
proc_lsst.multi INFO: found job 16966532 in provider escience_a40
2024-03-09 17:39:37 proc_lsst.multi:201 [INFO]  cancelling 16966532 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966532 on provider escience_a40
2024-03-09 17:39:37 proc_lsst.multi:146 [INFO]  found job 16966533 in provider escience_a40
proc_lsst.multi INFO: found job 16966533 in provider escience_a40
2024-03-09 17:39:37 proc_lsst.multi:201 [INFO]  cancelling 16966533 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966533 on provider escience_a40
2024-03-09 17:39:38 proc_lsst.multi:146 [INFO]  found job 16966534 in provider escience_a40
proc_lsst.multi INFO: found job 16966534 in provider escience_a40
2024-03-09 17:39:38 proc_lsst.multi:201 [INFO]  cancelling 16966534 on provider escience_a40
proc_lsst.multi INFO: cancelling 16966534 on provider escience_a40
2024-03-09 17:39:38 proc_lsst.multi:146 [INFO]  found job 16966510 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966510 in provider escience_rtx6k
2024-03-09 17:39:38 proc_lsst.multi:201 [INFO]  cancelling 16966510 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16966510 on provider escience_rtx6k
2024-03-09 17:39:38 proc_lsst.multi:146 [INFO]  found job 16966511 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966511 in provider escience_rtx6k
2024-03-09 17:39:38 proc_lsst.multi:201 [INFO]  cancelling 16966511 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16966511 on provider escience_rtx6k
2024-03-09 17:39:38 proc_lsst.multi:146 [INFO]  found job 16966512 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966512 in provider escience_rtx6k
2024-03-09 17:39:38 proc_lsst.multi:201 [INFO]  cancelling 16966512 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16966512 on provider escience_rtx6k
2024-03-09 17:39:38 proc_lsst.multi:146 [INFO]  found job 16966513 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966513 in provider escience_rtx6k
2024-03-09 17:39:38 proc_lsst.multi:201 [INFO]  cancelling 16966513 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16966513 on provider escience_rtx6k
2024-03-09 17:39:38 proc_lsst.multi:146 [INFO]  found job 16966514 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16966514 in provider escience_rtx6k
2024-03-09 17:39:38 proc_lsst.multi:201 [INFO]  cancelling 16966514 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16966514 on provider escience_rtx6k
2024-03-09 17:39:38 proc_lsst.multi:146 [INFO]  found job 64340 in provider local
proc_lsst.multi INFO: found job 64340 in provider local
2024-03-09 17:39:38 proc_lsst.multi:201 [INFO]  cancelling 64340 on provider local
proc_lsst.multi INFO: cancelling 64340 on provider local
parsl.providers.local.local DEBUG: Terminating job/proc_id: 64340
parsl.providers.local.local WARNING: Failed to kill PID: 64340 and child processes on local
2024-03-09 17:39:40 proc_lsst.multi:50 [INFO]  no new jobs since last cancel, resuming executor shutdown
proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown
parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown
parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt
parsl.dataflow.dflow INFO: Shut down executor multi
parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal
parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete
parsl.executors.threads DEBUG: Done with executor shutdown
parsl.dataflow.dflow INFO: Shut down executor _parsl_internal
parsl.dataflow.dflow INFO: Terminated executors
parsl.dataflow.dflow INFO: DFK cleanup complete
parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread
lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 244.3569 seconds
lsst.ctrl.bps.drivers INFO: Run 'DEEP_20190507_A0c_science#step0_20240309T173428Z' submitted for execution with id 'None'
lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 244.3680 seconds; current memory usage: 0.279 Gibyte, delta: 0.005 Gibyte, peak delta: 0.007 Gibyte
lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 291.4398 seconds; current memory usage: 0.279 Gibyte, delta: 0.142 Gibyte, peak delta: 0.144 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.281 Gibyte (main), 0.281 Gibyte (largest child process)
Run Id: None
Run Name: DEEP_20190507_A0c_science#step0_20240309T173428Z
parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up

real	5m7.344s
user	1m24.128s
sys	0m44.791s