+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20210516/A0g --output-run DEEP/20210516/A0g/science#step0/20240315T224631Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 3.7686 seconds; current memory usage: 0.179 Gibyte, delta: 0.042 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0041 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0076 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/pipeline_orig.qgraph DEEP/20210516/A0g/science#step0/20240315T224631Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 2.8993 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 7.8655 seconds; current memory usage: 0.272 Gibyte, delta: 0.093 Gibyte, peak delta: 0.082 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.265 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0147 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.265 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 372 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20210516_A0g_science#step0_20240315T224631Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.0930 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.265 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 374 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.2600 seconds; current memory usage: 0.274 Gibyte, delta: 0.001 Gibyte, peak delta: 0.001 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.274 Gibyte (main), 0.265 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: ab52447d-3e53-4fa4-8ae4-055a6ca1e548 parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-15 22:46:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-15 22:46:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:46:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-15 22:46:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:46:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-15 22:46:58 proc_lsst.multi:170 [INFO] [multi] job_id 36385 proc_lsst.multi INFO: [multi] job_id 36385 2024-03-15 22:46:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:46:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:46:58 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 36385 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 36385 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 36385 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash e42459fc873592769083f016a1c7faaa parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a559300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/07c43253-3830-4a24-a51c-e01077a3ceda_isrForCrosstalkSources_998212_28.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/07c43253-3830-4a24-a51c-e01077a3ceda_isrForCrosstalkSources_998212_28.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash d9616d5ee877201e2da8e63ac897ea9e parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a559620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/4b890ce5-21fa-4544-a798-2e6abb58c5d2_isrForCrosstalkSources_998210_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/4b890ce5-21fa-4544-a798-2e6abb58c5d2_isrForCrosstalkSources_998210_57.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 89b42038f090c0f79e1f52a364ea4de8 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a559940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/42ef80d4-e9aa-4eb8-9a14-49baa9be223a_isrForCrosstalkSources_998106_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/42ef80d4-e9aa-4eb8-9a14-49baa9be223a_isrForCrosstalkSources_998106_30.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash fde8a4728c18950b75c2adabbd405b7d parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a559800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/e8da4bb8-ae6d-43e3-8bc9-840e356a48be_isrForCrosstalkSources_998211_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/e8da4bb8-ae6d-43e3-8bc9-840e356a48be_isrForCrosstalkSources_998211_57.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 9559faff1c9f8d4cf47314eff9628fa4 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a559a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/c584b445-996a-4b11-a3e8-c66a971bdf5a_isrForCrosstalkSources_998106_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/c584b445-996a-4b11-a3e8-c66a971bdf5a_isrForCrosstalkSources_998106_55.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 3c71d5e25d11c54b9ad899516f2b2ba7 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5598a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/b9776b41-3387-4142-ac99-68d81ecafde2_isrForCrosstalkSources_998212_61.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/b9776b41-3387-4142-ac99-68d81ecafde2_isrForCrosstalkSources_998212_61.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 62c374128f647751be08e92df88b18a6 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a559b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/491f0ccf-51a9-476c-bd8c-1dc89f5114b1_isrForCrosstalkSources_998106_46.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/491f0ccf-51a9-476c-bd8c-1dc89f5114b1_isrForCrosstalkSources_998106_46.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash fb1a5dc50e5531e1f26c0c81bd4323b3 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a558c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/1463b5cb-768c-401c-87bb-5a84fefb2744_isrForCrosstalkSources_998106_25.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/1463b5cb-768c-401c-87bb-5a84fefb2744_isrForCrosstalkSources_998106_25.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 54bf201f70cc452b1a2270dd7e2f9f1a parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a559d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/208fb8dd-2f78-4109-bd3c-4345b9a56fdf_isrForCrosstalkSources_998212_21.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/208fb8dd-2f78-4109-bd3c-4345b9a56fdf_isrForCrosstalkSources_998212_21.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash ff5cde6d01021c229eceb14d30e21222 parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5594e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/fe72e78a-1580-41d7-a079-c3016a31193d_isrForCrosstalkSources_998212_60.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/fe72e78a-1580-41d7-a079-c3016a31193d_isrForCrosstalkSources_998212_60.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 4abc58e7db00a5ca967f1f34d9d613d7 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5599e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/0a43a729-7352-4685-a26a-3d533aa3fc16_isrForCrosstalkSources_998212_8.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/0a43a729-7352-4685-a26a-3d533aa3fc16_isrForCrosstalkSources_998212_8.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash c96e0ace76eb6d92dcad2a0b1e143dc8 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a559c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/57fc39b6-abb5-48f3-b49c-e0cc60da6897_isrForCrosstalkSources_998108_38.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/57fc39b6-abb5-48f3-b49c-e0cc60da6897_isrForCrosstalkSources_998108_38.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 7002ef7f26b1856c33e79dc43a4d5845 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a559f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/cbe910f3-3c43-4025-beae-e986943ef6ec_isrForCrosstalkSources_998108_26.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/cbe910f3-3c43-4025-beae-e986943ef6ec_isrForCrosstalkSources_998108_26.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash d638136d663b70c4655c2b36ad4369ba parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/aa615a90-8aa8-42ce-a8d9-ebbbd8b399be_isrForCrosstalkSources_998108_37.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/aa615a90-8aa8-42ce-a8d9-ebbbd8b399be_isrForCrosstalkSources_998108_37.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 37d9b2051880121a761dffc13a7fe53b parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/076f97fb-11ce-4a56-b71e-aece0a6fcda8_isrForCrosstalkSources_998211_62.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/076f97fb-11ce-4a56-b71e-aece0a6fcda8_isrForCrosstalkSources_998211_62.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 4fdaf1d23b941edba8627fca38e2ffa4 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a559ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/aaa9a1d9-8619-4d67-ae24-4000247904f7_isrForCrosstalkSources_998210_18.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/aaa9a1d9-8619-4d67-ae24-4000247904f7_isrForCrosstalkSources_998210_18.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash e6220f105075b611c2b868190c7532cb parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/6f041b6f-b1fc-4ed5-8a57-115d3398399b_isrForCrosstalkSources_998107_20.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/6f041b6f-b1fc-4ed5-8a57-115d3398399b_isrForCrosstalkSources_998107_20.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 08bcabe048746f733c623e17ee566920 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/1a870a33-df08-4718-82fb-8ce6e7faef9b_isrForCrosstalkSources_998106_27.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/1a870a33-df08-4718-82fb-8ce6e7faef9b_isrForCrosstalkSources_998106_27.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash c825007bdc92f0162dc45ec81556e0e9 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/d782e0c0-a046-44ce-a435-73d2aedaa1c3_isrForCrosstalkSources_998211_6.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/d782e0c0-a046-44ce-a435-73d2aedaa1c3_isrForCrosstalkSources_998211_6.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash a28ae174b17b37086690ff0841300c3b parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/7bb75482-bccc-45a4-acbb-e0c383d2e336_isrForCrosstalkSources_998108_12.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/7bb75482-bccc-45a4-acbb-e0c383d2e336_isrForCrosstalkSources_998108_12.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash ad53e568cab260599ab232719993de45 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/66e59bf4-6dee-4246-92ec-963029ae362f_isrForCrosstalkSources_998211_37.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/66e59bf4-6dee-4246-92ec-963029ae362f_isrForCrosstalkSources_998211_37.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 9c4fa9feca951231fa2f03f9e655042b parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/104648bd-b818-4c56-877d-8d738fc195c7_isrForCrosstalkSources_998107_58.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/104648bd-b818-4c56-877d-8d738fc195c7_isrForCrosstalkSources_998107_58.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 332f383fed125d439b89bdd88c5a5bbd parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/c56b6e7a-2a6b-47fd-b407-5eed9200b065_isrForCrosstalkSources_998106_54.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/c56b6e7a-2a6b-47fd-b407-5eed9200b065_isrForCrosstalkSources_998106_54.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 926bcfe3ae80a9052987669e45543be3 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/241954d5-1a3c-444f-971f-5be30acb5e08_isrForCrosstalkSources_998108_16.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/241954d5-1a3c-444f-971f-5be30acb5e08_isrForCrosstalkSources_998108_16.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 4177a54f59de07d813c035eb5d9993d3 parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/57bb81d5-5d22-4142-b614-12efca3cb663_isrForCrosstalkSources_998212_40.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/57bb81d5-5d22-4142-b614-12efca3cb663_isrForCrosstalkSources_998212_40.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 54c9ef240cc917899e78b6963af6d0bd parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/7f33b8b1-f5f9-47c0-a407-7b2a7be3126c_isrForCrosstalkSources_998212_23.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/7f33b8b1-f5f9-47c0-a407-7b2a7be3126c_isrForCrosstalkSources_998212_23.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 9fdf4f7a1a3ac9fd1c2ae717a9680e90 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/3c99bca7-cb56-4b2c-8c9b-5fd2d246ef1f_isrForCrosstalkSources_998210_6.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/3c99bca7-cb56-4b2c-8c9b-5fd2d246ef1f_isrForCrosstalkSources_998210_6.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 575fc68a6bc63dd60428f38c0cebc726 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/6614a1a1-a992-46b4-923e-ef703911d2d7_isrForCrosstalkSources_998212_57.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/6614a1a1-a992-46b4-923e-ef703911d2d7_isrForCrosstalkSources_998212_57.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 710c11135c8d3cb40c0e585917696a92 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/739349c5-118c-4f9b-8410-273019e5939f_isrForCrosstalkSources_998211_21.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/739349c5-118c-4f9b-8410-273019e5939f_isrForCrosstalkSources_998211_21.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 17fbdd1532b20e23b605984bc49b29f3 parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/0405daba-c5f4-4629-91cd-3bc077fd074f_isrForCrosstalkSources_998211_60.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/0405daba-c5f4-4629-91cd-3bc077fd074f_isrForCrosstalkSources_998211_60.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 228ab7474024a6503e2699fbc1726098 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/240e0c3a-d4f4-4f1c-8964-57acf74e37af_isrForCrosstalkSources_998211_36.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/240e0c3a-d4f4-4f1c-8964-57acf74e37af_isrForCrosstalkSources_998211_36.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 57858a26bba8b9f07f7fb0efa8173747 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/9aaa9385-85fa-42b6-b978-432478a14acc_isrForCrosstalkSources_998107_14.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/9aaa9385-85fa-42b6-b978-432478a14acc_isrForCrosstalkSources_998107_14.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash d34a242d716737794bf6eab3caf99f55 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/1f6c135f-5bb2-446e-af75-531df35c5532_isrForCrosstalkSources_998108_17.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/1f6c135f-5bb2-446e-af75-531df35c5532_isrForCrosstalkSources_998108_17.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash f9431c9e3d6cbb7f6548505f4069757a parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/736e5649-8b6d-4daf-b44a-817750c50b22_isrForCrosstalkSources_998211_44.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/736e5649-8b6d-4daf-b44a-817750c50b22_isrForCrosstalkSources_998211_44.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash df3494e46386be418fc0f4619c39ac2a parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/fe0051eb-efec-4d10-b2c8-2b5c4f4cf7f7_isrForCrosstalkSources_998108_42.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/fe0051eb-efec-4d10-b2c8-2b5c4f4cf7f7_isrForCrosstalkSources_998108_42.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash b4e97170402f319ce24e71e420f09f34 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/0c3c6df5-69e3-4cfd-94f1-64d87ab2d0d5_isrForCrosstalkSources_998211_26.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/0c3c6df5-69e3-4cfd-94f1-64d87ab2d0d5_isrForCrosstalkSources_998211_26.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 4f7a95146196668b42d043481cd426d9 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/d0197f6d-d65e-4876-a16b-f73d0000e4e0_isrForCrosstalkSources_998108_34.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/d0197f6d-d65e-4876-a16b-f73d0000e4e0_isrForCrosstalkSources_998108_34.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 1d00711cfadfd124dc7c24a4b40864c7 parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/a821e10d-d905-4ea7-abcc-0e091dbbdb62_isrForCrosstalkSources_998212_14.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/a821e10d-d905-4ea7-abcc-0e091dbbdb62_isrForCrosstalkSources_998212_14.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 8e6fc8bffa740bc18200c9cbfb0af2c5 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/8445c22c-996f-432b-8aec-2e91f090df51_isrForCrosstalkSources_998212_2.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/8445c22c-996f-432b-8aec-2e91f090df51_isrForCrosstalkSources_998212_2.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 5290962272b53531e9731bf08c2d8c67 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/1396534f-d8aa-4c9c-bf85-329d126a2be7_isrForCrosstalkSources_998211_40.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/1396534f-d8aa-4c9c-bf85-329d126a2be7_isrForCrosstalkSources_998211_40.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash ae7f052d18eb9bb58508bb95950339cc parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/dc720944-adf2-4203-a00d-68c85cce107d_isrForCrosstalkSources_998211_12.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/dc720944-adf2-4203-a00d-68c85cce107d_isrForCrosstalkSources_998211_12.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 5f91987a2ebf3869b079392f56639954 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/5d09b665-b87f-439a-ba25-0b2c4e063bf2_isrForCrosstalkSources_998210_31.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/5d09b665-b87f-439a-ba25-0b2c4e063bf2_isrForCrosstalkSources_998210_31.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 68d28dd051d23243639caf0cf27e1399 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/ee0f82c6-2a80-4462-86cc-325c0e534d92_isrForCrosstalkSources_998107_15.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/ee0f82c6-2a80-4462-86cc-325c0e534d92_isrForCrosstalkSources_998107_15.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash f099a69c640458a4912d67c847022504 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/e13c21f3-e94c-464e-9e89-bbd2d8bef640_isrForCrosstalkSources_998107_42.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/e13c21f3-e94c-464e-9e89-bbd2d8bef640_isrForCrosstalkSources_998107_42.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 95b14a1391a54374cf6ea1ac126429f5 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/f72e96ff-db9d-4a93-ab0d-c4be135d9c1a_isrForCrosstalkSources_998212_58.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/f72e96ff-db9d-4a93-ab0d-c4be135d9c1a_isrForCrosstalkSources_998212_58.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 46937c8feae7676e52a95c347931b774 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/a9125b79-5edc-46b4-be38-4ce143685b26_isrForCrosstalkSources_998107_38.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/a9125b79-5edc-46b4-be38-4ce143685b26_isrForCrosstalkSources_998107_38.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 7db6ba69437e17ed2ca0bc99e3c92294 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/432e24a4-a756-4d10-a6f4-d2d01c3ff85f_isrForCrosstalkSources_998107_49.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/432e24a4-a756-4d10-a6f4-d2d01c3ff85f_isrForCrosstalkSources_998107_49.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 598ea1e338023595ebdcd551199b44d8 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/32c1a5ff-405b-4220-9f81-b492482167ea_isrForCrosstalkSources_998106_11.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/32c1a5ff-405b-4220-9f81-b492482167ea_isrForCrosstalkSources_998106_11.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 34db750c55d985dcc787fe2e98e0c8fc parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/123ece1c-5ce9-42cd-8dd5-9f76d3baba8f_isrForCrosstalkSources_998108_30.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/123ece1c-5ce9-42cd-8dd5-9f76d3baba8f_isrForCrosstalkSources_998108_30.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash ff45a362f88b2e684487b8722196fe62 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/feb19a91-5677-4f6a-af49-37db96185d72_isrForCrosstalkSources_998211_17.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/feb19a91-5677-4f6a-af49-37db96185d72_isrForCrosstalkSources_998211_17.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 182afa13e1b29e77ed82e2c374bdfb9f parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/eabeb3a5-b28a-411f-9956-45aac436d7e1_isrForCrosstalkSources_998107_34.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/eabeb3a5-b28a-411f-9956-45aac436d7e1_isrForCrosstalkSources_998107_34.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash d750e7438e5ff52f24df2c2bc86800b1 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/e6fde39c-fedd-45ef-83f3-8cd39042c96d_isrForCrosstalkSources_998108_2.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/e6fde39c-fedd-45ef-83f3-8cd39042c96d_isrForCrosstalkSources_998108_2.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash b0ce7ccb71eb56661015a5721fb3fb9b parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/c31e2e4b-5fac-4a27-92d8-91c61ca7bfb9_isrForCrosstalkSources_998210_40.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/c31e2e4b-5fac-4a27-92d8-91c61ca7bfb9_isrForCrosstalkSources_998210_40.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 943bcb219d9bc088b2461b4d09ca778d parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/3c4b2933-f7ea-46b4-819e-77cd0ec2d373_isrForCrosstalkSources_998107_12.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/3c4b2933-f7ea-46b4-819e-77cd0ec2d373_isrForCrosstalkSources_998107_12.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 48ce2107539428e165eff4b36f7434e3 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/51bc0899-2611-4ad2-8c2e-3b60f9d73a83_isrForCrosstalkSources_998106_17.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/51bc0899-2611-4ad2-8c2e-3b60f9d73a83_isrForCrosstalkSources_998106_17.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 9fa0a17f628e146a46f1b38064c3520e parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/5e45f162-54b7-42e2-b7b9-dc9de54a91fe_isrForCrosstalkSources_998106_33.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/5e45f162-54b7-42e2-b7b9-dc9de54a91fe_isrForCrosstalkSources_998106_33.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 1d44c28e884f766ea14978f3360de2e3 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/c9124bf5-7869-4800-ab3a-c30b988a18f6_isrForCrosstalkSources_998108_29.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/c9124bf5-7869-4800-ab3a-c30b988a18f6_isrForCrosstalkSources_998108_29.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash e6f15f8f34c41e35b5e70d0adc12c13b parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/c2a47194-c054-4a8c-aa53-96868430049d_isrForCrosstalkSources_998106_5.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/c2a47194-c054-4a8c-aa53-96868430049d_isrForCrosstalkSources_998106_5.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 6bc759a6d3df22a1a7bf272ad53eeba5 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/8e294298-b1a9-4c93-8d35-6cd985a20475_isrForCrosstalkSources_998108_40.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/8e294298-b1a9-4c93-8d35-6cd985a20475_isrForCrosstalkSources_998108_40.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 987f1b3ccb948c99e44ea63b0bc18e9f parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/9e098689-2431-40ba-bc5d-e98514fdbb4e_isrForCrosstalkSources_998211_18.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/9e098689-2431-40ba-bc5d-e98514fdbb4e_isrForCrosstalkSources_998211_18.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 56705a745b50fd0ecd74022f9e8c94e1 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a55a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/d436dac6-bbc1-491d-a5d7-d112d0450032_isrForCrosstalkSources_998212_1.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/d436dac6-bbc1-491d-a5d7-d112d0450032_isrForCrosstalkSources_998212_1.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash ac698e69abc75ab03a2eb79ff0379590 parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5940e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/3df08b00-0815-48f9-942a-a732b26f2985_isrForCrosstalkSources_998107_56.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/3df08b00-0815-48f9-942a-a732b26f2985_isrForCrosstalkSources_998107_56.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash e9e33010f7db2f311b77176941f1b8de parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a594180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/3cc721a9-33eb-4577-9e03-d70522f4f59b_isrForCrosstalkSources_998106_51.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/3cc721a9-33eb-4577-9e03-d70522f4f59b_isrForCrosstalkSources_998106_51.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash e23a50ae31ac6d27865e9126df5087e2 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5942c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/a2cf66dc-22e1-46d0-9a19-ba7fbdfb32c7_isrForCrosstalkSources_998210_28.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/a2cf66dc-22e1-46d0-9a19-ba7fbdfb32c7_isrForCrosstalkSources_998210_28.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 739d0fcda6500aacb7e9197624297f59 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a594220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/00dde2fb-a720-41b3-a799-48c63c3bfbd7_isrForCrosstalkSources_998211_20.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/00dde2fb-a720-41b3-a799-48c63c3bfbd7_isrForCrosstalkSources_998211_20.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 9b3ae7a03b5a787beb012658539901b1 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a594360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/6db42cf2-30a9-4419-ac60-aacfaa9ae88d_isrForCrosstalkSources_998106_35.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/6db42cf2-30a9-4419-ac60-aacfaa9ae88d_isrForCrosstalkSources_998106_35.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 9542a8f00d25610a48723229ae3cdde1 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a594400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/08467930-8d13-4950-b1f2-b523d10e5ef2_isrForCrosstalkSources_998211_41.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/08467930-8d13-4950-b1f2-b523d10e5ef2_isrForCrosstalkSources_998211_41.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash b209aca12f04d9bab7a0a90d297f132a parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5944a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/fdaf5b46-b707-4b65-b97b-056168a3860c_isrForCrosstalkSources_998212_37.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/fdaf5b46-b707-4b65-b97b-056168a3860c_isrForCrosstalkSources_998212_37.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 212067eb3e2a330ddf90f7b8f119eb67 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5945e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/f7c6fe07-68e9-4aa2-9706-e3d5632ffc33_isrForCrosstalkSources_998108_33.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/f7c6fe07-68e9-4aa2-9706-e3d5632ffc33_isrForCrosstalkSources_998108_33.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 04388e3c2c337becf711d27501bb518f parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a594540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/37426948-d34c-4598-9ed8-3c3244e02cca_isrForCrosstalkSources_998210_59.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/37426948-d34c-4598-9ed8-3c3244e02cca_isrForCrosstalkSources_998210_59.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 0c47bd144017353fb83578736a510bc1 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a594680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/affe2d51-1963-4da2-8404-a64028804c68_isrForCrosstalkSources_998108_9.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/affe2d51-1963-4da2-8404-a64028804c68_isrForCrosstalkSources_998108_9.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 4dc1f4f22e86899342e6b5fa9e1acbb6 parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a594720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/9c34cc71-1a6f-4755-8e12-8914d3ae06da_isrForCrosstalkSources_998210_37.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/9c34cc71-1a6f-4755-8e12-8914d3ae06da_isrForCrosstalkSources_998210_37.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 251595125f7a563e92d656ca9b8bc11f parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5947c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/9ad2a95c-8330-44b1-af42-c11b0d6b83fc_isrForCrosstalkSources_998108_24.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/9ad2a95c-8330-44b1-af42-c11b0d6b83fc_isrForCrosstalkSources_998108_24.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 6a63d3bc0843b88a1d66a079a31c2019 parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a594900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/f304118b-ab79-458e-8a4a-e5b52bd5ad0f_isrForCrosstalkSources_998210_32.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/f304118b-ab79-458e-8a4a-e5b52bd5ad0f_isrForCrosstalkSources_998210_32.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 55413e969c45b4bc2b4ff67fad70eed3 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a594860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/39d6521c-c545-4c4e-a1a6-febf76f5b744_isrForCrosstalkSources_998108_54.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/39d6521c-c545-4c4e-a1a6-febf76f5b744_isrForCrosstalkSources_998108_54.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash ec7f677d0f5f2ce4e976d60b1e71b476 parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5949a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/f337e158-b091-4d37-9990-7da49cfbafcf_isrForCrosstalkSources_998108_52.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/f337e158-b091-4d37-9990-7da49cfbafcf_isrForCrosstalkSources_998108_52.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 6503b46178d48405483274604c2be45c parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a594a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/e0da7857-942a-4d3e-8bba-c1bf20444fbf_isrForCrosstalkSources_998108_10.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/e0da7857-942a-4d3e-8bba-c1bf20444fbf_isrForCrosstalkSources_998108_10.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash 075c3200cceaeae485550159c5f3b83e parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a594ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/1c84e32b-b9f1-4814-8fdc-dafd4fff9944_isrForCrosstalkSources_998210_61.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/1c84e32b-b9f1-4814-8fdc-dafd4fff9944_isrForCrosstalkSources_998210_61.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 3c27439af42ec9dc901b2e9d274b35b7 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a594c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/87470ff5-320b-4af3-95da-144469ef3a3a_isrForCrosstalkSources_998107_29.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/87470ff5-320b-4af3-95da-144469ef3a3a_isrForCrosstalkSources_998107_29.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 534ce214e567ea92f8425f7fcdfe60f1 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a594b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/57a9cade-247a-4418-a309-f4c02379e11d_isrForCrosstalkSources_998107_50.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/57a9cade-247a-4418-a309-f4c02379e11d_isrForCrosstalkSources_998107_50.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 8ee2a7a0598170ca63a1c99f51424796 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a594cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/bc40e7c6-5a67-4cd3-83f2-48929377db30_isrForCrosstalkSources_998108_21.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/bc40e7c6-5a67-4cd3-83f2-48929377db30_isrForCrosstalkSources_998108_21.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 0f7c3128908f8a72f83437b5a532f552 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a594d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/06d3440b-29d2-41c3-b6a8-d00c22a3e3eb_isrForCrosstalkSources_998108_3.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/06d3440b-29d2-41c3-b6a8-d00c22a3e3eb_isrForCrosstalkSources_998108_3.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash effb9d6b50ce547400349b3317fc64d2 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a594e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/7c5f999d-4d9c-427b-9b59-cb6c3cf58616_isrForCrosstalkSources_998108_7.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/7c5f999d-4d9c-427b-9b59-cb6c3cf58616_isrForCrosstalkSources_998108_7.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 0ed2b9d1b6f521a388be371a0d4e2491 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a594f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/3daec035-0a91-4e21-87f2-d5f99457c8fd_isrForCrosstalkSources_998108_31.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/3daec035-0a91-4e21-87f2-d5f99457c8fd_isrForCrosstalkSources_998108_31.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash e2f17ea90509ea557555a5da5700c7ca parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a594ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/91bf3645-178e-47d0-9344-fcddeaef5376_isrForCrosstalkSources_998211_9.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/91bf3645-178e-47d0-9344-fcddeaef5376_isrForCrosstalkSources_998211_9.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash ca39e629f408138cf15acbc1a1d2ce84 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a594fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/f520da5e-f86e-4061-b873-48650dbf26da_isrForCrosstalkSources_998108_57.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/f520da5e-f86e-4061-b873-48650dbf26da_isrForCrosstalkSources_998108_57.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 113308a31c55166e42cd170f27943c07 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a595080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/422903b9-71ed-4db6-83dd-762e5d8dec71_isrForCrosstalkSources_998106_28.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/422903b9-71ed-4db6-83dd-762e5d8dec71_isrForCrosstalkSources_998106_28.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 82ab308e301a994fb96dcf04b1b6097c parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a595120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/7d5f5af2-ce71-4888-9240-0c3e54722dd6_isrForCrosstalkSources_998210_62.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/7d5f5af2-ce71-4888-9240-0c3e54722dd6_isrForCrosstalkSources_998210_62.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 788470ab9c5f29533fe372906c833ee1 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a595260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/debf6e4d-b555-4ec7-ab05-f14219b243c5_isrForCrosstalkSources_998107_43.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/debf6e4d-b555-4ec7-ab05-f14219b243c5_isrForCrosstalkSources_998107_43.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 12c07fa0383e9ececec9366342e81ce7 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5951c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/d7addd36-ae0a-444d-b21b-f2b064a00fa6_isrForCrosstalkSources_998210_47.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/d7addd36-ae0a-444d-b21b-f2b064a00fa6_isrForCrosstalkSources_998210_47.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 75e1392eba1f7befa003b982d274a1f6 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a595300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/29c55c5b-1650-4e24-b945-ebe0351a2920_isrForCrosstalkSources_998108_1.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/29c55c5b-1650-4e24-b945-ebe0351a2920_isrForCrosstalkSources_998108_1.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash cf03ec8ed1bb3e6ad82a801a7776373c parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5953a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/585a094e-0fc5-404c-a373-d188697947d9_isrForCrosstalkSources_998210_9.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/585a094e-0fc5-404c-a373-d188697947d9_isrForCrosstalkSources_998210_9.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 337c152739abaf775a97d32054730e0d parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a595440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/9fc78d1c-5d2f-4035-8ed2-d967b4e8a081_isrForCrosstalkSources_998108_60.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/9fc78d1c-5d2f-4035-8ed2-d967b4e8a081_isrForCrosstalkSources_998108_60.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 15d76a7fc7065e5a0e6169f9fd42ae34 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a595580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/ff92261f-871f-4d48-9e17-4ace669467d1_isrForCrosstalkSources_998210_41.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/ff92261f-871f-4d48-9e17-4ace669467d1_isrForCrosstalkSources_998210_41.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 5cea1bbaab454ef1bd2cdde478d576c1 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5954e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/ee88bf76-94b1-485f-944b-57b15eef8f23_isrForCrosstalkSources_998108_11.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/ee88bf76-94b1-485f-944b-57b15eef8f23_isrForCrosstalkSources_998108_11.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash f978cafa84ddaa8d644bd1abe0258163 parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a595620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/a564c785-470e-4b78-80e5-a6fa7b789e70_isrForCrosstalkSources_998108_5.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/a564c785-470e-4b78-80e5-a6fa7b789e70_isrForCrosstalkSources_998108_5.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash a672ee720c55832390a2763852b3d326 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5956c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/214320ca-92ca-4a81-9977-70b4cb27564f_isrForCrosstalkSources_998212_51.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/214320ca-92ca-4a81-9977-70b4cb27564f_isrForCrosstalkSources_998212_51.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 092784642a6133a0d2c58b08fee06c45 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a595760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/1a110039-2602-47ea-9e91-9331f3c2630d_isrForCrosstalkSources_998106_1.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/1a110039-2602-47ea-9e91-9331f3c2630d_isrForCrosstalkSources_998106_1.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 2ddf68a7adb2e948db77fa2840c49f81 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5958a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/ad533053-e53f-4c11-a420-2b26a5d701b2_isrForCrosstalkSources_998106_58.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/ad533053-e53f-4c11-a420-2b26a5d701b2_isrForCrosstalkSources_998106_58.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash b3179575c5b9d10b4974a93959c9c859 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a595800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/522a237e-ed78-4a64-9916-7e66fb65b732_isrForCrosstalkSources_998211_5.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/522a237e-ed78-4a64-9916-7e66fb65b732_isrForCrosstalkSources_998211_5.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 5787e21d6ea2dd20970b904117b246a1 parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a595940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/fced1cea-b533-41b7-afa3-896d703e2471_isrForCrosstalkSources_998212_35.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/fced1cea-b533-41b7-afa3-896d703e2471_isrForCrosstalkSources_998212_35.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash d18ed09f55e668d67d9c34ee0e0dc164 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5959e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/fce460bc-9311-4b23-86fd-fff8466ac91a_isrForCrosstalkSources_998107_10.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/fce460bc-9311-4b23-86fd-fff8466ac91a_isrForCrosstalkSources_998107_10.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 8eac15ea74dde0deeb4d722a1b4f8701 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a595a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/9f04b63f-3c84-4a58-b608-3fec2b2caf8b_isrForCrosstalkSources_998210_51.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/9f04b63f-3c84-4a58-b608-3fec2b2caf8b_isrForCrosstalkSources_998210_51.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 03844d7e47d4f24d1fe798d6c3b6def2 parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a595bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/2a8ab7d8-1fd1-44f0-95a3-956be18dcaec_isrForCrosstalkSources_998210_44.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/2a8ab7d8-1fd1-44f0-95a3-956be18dcaec_isrForCrosstalkSources_998210_44.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash e78dff2a4ee70f91f5ebac5d93ce086b parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a595b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/17e25ac4-9de9-4fb3-a8d3-b834dd3604d8_isrForCrosstalkSources_998210_33.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/17e25ac4-9de9-4fb3-a8d3-b834dd3604d8_isrForCrosstalkSources_998210_33.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash f86c14c16a8474055c3bc619facea84c parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a595c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/6c608334-41ec-4e7f-a0a1-d5d41318ae67_isrForCrosstalkSources_998107_44.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/6c608334-41ec-4e7f-a0a1-d5d41318ae67_isrForCrosstalkSources_998107_44.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 0c3c38da89039c3923883942055c77bb parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a595d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/aa5f56a1-4c0d-4af5-a7ee-39e344cc02aa_isrForCrosstalkSources_998210_43.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/aa5f56a1-4c0d-4af5-a7ee-39e344cc02aa_isrForCrosstalkSources_998210_43.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 33116c3f247609e0d4a54c3cec7eb622 parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a595da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/19b46b69-6563-4657-a5ad-6324f06f87c9_isrForCrosstalkSources_998107_23.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/19b46b69-6563-4657-a5ad-6324f06f87c9_isrForCrosstalkSources_998107_23.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash e6eda90e67851fe9cb5a7e83a67f9d07 parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a595ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/0cf24121-70bb-4fde-bdcf-400513ef8397_isrForCrosstalkSources_998108_47.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/0cf24121-70bb-4fde-bdcf-400513ef8397_isrForCrosstalkSources_998108_47.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash a599e18d86d8493e7a1fef5d35742296 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a595e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/118fd9bc-8331-43f0-9c7d-8017bd9a6308_isrForCrosstalkSources_998107_18.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/118fd9bc-8331-43f0-9c7d-8017bd9a6308_isrForCrosstalkSources_998107_18.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 31fca1102aa37a8bb5104de2ae9b5533 parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a595f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/5eed6233-ae15-4bf3-8c48-5db75b66d2de_isrForCrosstalkSources_998108_48.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/5eed6233-ae15-4bf3-8c48-5db75b66d2de_isrForCrosstalkSources_998108_48.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 38b64487d0cd0380993cba1ff2e5d360 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a596020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 22:46:59 proc_lsst.multi:146 [INFO] found job 36385 in provider local proc_lsst.multi INFO: found job 36385 in provider local parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/dc652169-c5b1-40a5-b1d3-9cf1b8f8053f_isrForCrosstalkSources_998211_23.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/dc652169-c5b1-40a5-b1d3-9cf1b8f8053f_isrForCrosstalkSources_998211_23.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 7a690f0b91c3cb4a95e2be53a9047cba parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5960c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/d1ee50e1-b5fd-4c50-afd4-642c22d94fec_isrForCrosstalkSources_998210_36.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/d1ee50e1-b5fd-4c50-afd4-642c22d94fec_isrForCrosstalkSources_998210_36.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash e8221d4eaffdc3570ebf766a9f5d39ba parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a596200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/36cbee7e-db74-46ea-b13e-3e1e64de8344_isrForCrosstalkSources_998212_43.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/36cbee7e-db74-46ea-b13e-3e1e64de8344_isrForCrosstalkSources_998212_43.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 545c2c306129447b07070f87b1ca423e parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a596160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/409a0357-3222-41fc-be24-6e2bcdd9cabb_isrForCrosstalkSources_998210_4.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/409a0357-3222-41fc-be24-6e2bcdd9cabb_isrForCrosstalkSources_998210_4.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash cc1e7515d58a61a039ba3bce5046e989 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a596480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/4b270735-285c-4d20-93f3-101a6abd99a0_isrForCrosstalkSources_998212_11.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/4b270735-285c-4d20-93f3-101a6abd99a0_isrForCrosstalkSources_998212_11.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 2f8325869a04b2ec5d656f908d9c6a88 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a596700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/36570a24-2920-412b-968d-5fc57645b155_isrForCrosstalkSources_998108_13.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/36570a24-2920-412b-968d-5fc57645b155_isrForCrosstalkSources_998108_13.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 4c6c51d288b278e74b706ac5a3e0bdda parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a596660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/83279349-983f-4a47-808e-dc084686d063_isrForCrosstalkSources_998210_56.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/83279349-983f-4a47-808e-dc084686d063_isrForCrosstalkSources_998210_56.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 6eca4e9283eef14c41552aa39add9dcb parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5965c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/760ef7a0-eb4a-4b4f-a407-7c399b09102b_isrForCrosstalkSources_998107_22.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/760ef7a0-eb4a-4b4f-a407-7c399b09102b_isrForCrosstalkSources_998107_22.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 902289b537c08173b75cb92e964024ab parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5967a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/0fe8964a-5225-4159-8582-b4573bc3d7fe_isrForCrosstalkSources_998211_24.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/0fe8964a-5225-4159-8582-b4573bc3d7fe_isrForCrosstalkSources_998211_24.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash fcc6ab635ddde15b933f929c94a1b554 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a596840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/8d4c9a89-3587-49f4-a29f-9e354571cb12_isrForCrosstalkSources_998107_55.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/8d4c9a89-3587-49f4-a29f-9e354571cb12_isrForCrosstalkSources_998107_55.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 9c23fb274b3c35a01e1acea4f3705ed5 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5968e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/ab2e173d-79ea-402c-9582-39d2d012f14d_isrForCrosstalkSources_998212_48.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/ab2e173d-79ea-402c-9582-39d2d012f14d_isrForCrosstalkSources_998212_48.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 7ab692b8a6646bb94d5e8d886496a40f parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a596980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/e5dc846d-8bcb-4621-a396-c53d3e89d197_isrForCrosstalkSources_998211_32.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/e5dc846d-8bcb-4621-a396-c53d3e89d197_isrForCrosstalkSources_998211_32.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 260a52764ae8a60de01c0e2816077f2b parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a596b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/89014e12-3857-4c8d-a9c8-6a1fe33bfe54_isrForCrosstalkSources_998210_45.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/89014e12-3857-4c8d-a9c8-6a1fe33bfe54_isrForCrosstalkSources_998210_45.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 870515d3c291fe7c4b09afd04ccaac7f parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a596ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/50db07e7-2f54-4424-bd39-afc66f8089ab_isrForCrosstalkSources_998212_29.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/50db07e7-2f54-4424-bd39-afc66f8089ab_isrForCrosstalkSources_998212_29.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash ed66422a81d15f3de794b93f5c899da0 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a596a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/6382dd28-b59e-4ad1-badd-19f1e2692445_isrForCrosstalkSources_998106_16.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/6382dd28-b59e-4ad1-badd-19f1e2692445_isrForCrosstalkSources_998106_16.stderr parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 4928207018a826351faab84ec5074c64 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a596c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/9cf3e5f7-9c5e-4d0f-9c28-8099debaca49_isrForCrosstalkSources_998211_16.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/9cf3e5f7-9c5e-4d0f-9c28-8099debaca49_isrForCrosstalkSources_998211_16.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 3e529c5df18e307d62f494507b179f6c parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a596ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/d9b6d7e2-a1f2-431c-b9fa-e0176fa341d5_isrForCrosstalkSources_998108_25.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/d9b6d7e2-a1f2-431c-b9fa-e0176fa341d5_isrForCrosstalkSources_998108_25.stderr parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash e71eb3484583b0555b73b7eb94c872e2 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a596e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/91c8d1e6-abcb-4b2f-9a81-75791fe39154_isrForCrosstalkSources_998212_39.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/91c8d1e6-abcb-4b2f-9a81-75791fe39154_isrForCrosstalkSources_998212_39.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash b9fec743011c49f8ed7c52e6ada08ce9 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a596d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/4e770f1a-16c0-4834-9776-79e1e749b941_isrForCrosstalkSources_998108_56.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/4e770f1a-16c0-4834-9776-79e1e749b941_isrForCrosstalkSources_998108_56.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash d5d4fed304c5c560aca24f68c9044829 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a596f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/7266f033-ec69-424b-9808-601ffabb3882_isrForCrosstalkSources_998107_11.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/7266f033-ec69-424b-9808-601ffabb3882_isrForCrosstalkSources_998107_11.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 5a5a4571c012edb36556ebaf5315f674 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a596fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/54860232-d4fc-484c-90a6-b5857084f097_isrForCrosstalkSources_998212_25.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/54860232-d4fc-484c-90a6-b5857084f097_isrForCrosstalkSources_998212_25.stderr parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 2e8f73163ba2f4addbb6080bd759303d parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a596de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/46b81430-1a16-423f-a244-b3ddd199a103_isrForCrosstalkSources_998211_28.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/46b81430-1a16-423f-a244-b3ddd199a103_isrForCrosstalkSources_998211_28.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 2c2d4969d9cb4c81be3ea47335e36b9f parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5971a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/3a04beef-dfec-42ff-8a90-9a389982db0b_isrForCrosstalkSources_998212_46.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/3a04beef-dfec-42ff-8a90-9a389982db0b_isrForCrosstalkSources_998212_46.stderr parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash fd5741f9260c287e1868137c0caf3b6c parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a597060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/e373348b-6b49-4232-8936-24f50c3dc14a_isrForCrosstalkSources_998211_42.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/e373348b-6b49-4232-8936-24f50c3dc14a_isrForCrosstalkSources_998211_42.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash cd56d3ecba7c02edec8aaa5cea898d70 parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a597240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/69ce9ea8-6dbd-468e-8e83-dab8191b768c_isrForCrosstalkSources_998107_19.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/69ce9ea8-6dbd-468e-8e83-dab8191b768c_isrForCrosstalkSources_998107_19.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 207689f4fcb26abebd62921dfd8cfb3e parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5972e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/8b4d7144-d9ff-442d-9b10-30314082e196_isrForCrosstalkSources_998210_34.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/8b4d7144-d9ff-442d-9b10-30314082e196_isrForCrosstalkSources_998210_34.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 188f338ee44cea11e3c2f5d4f615a6f3 parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a597100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/7f178f00-ba08-4c4d-9807-ae47b508f09a_isrForCrosstalkSources_998107_31.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/7f178f00-ba08-4c4d-9807-ae47b508f09a_isrForCrosstalkSources_998107_31.stderr parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 610d238835ef50376d2637ce1b2dea02 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a597380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/3e17ec25-d0ab-469c-b804-440f5d67a633_isrForCrosstalkSources_998108_50.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/3e17ec25-d0ab-469c-b804-440f5d67a633_isrForCrosstalkSources_998108_50.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash b31ad07f7627094dc99138186d493dab parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a597420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/aabe4cae-e4a3-4fb4-b2c6-9440b3fa155b_isrForCrosstalkSources_998107_32.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/aabe4cae-e4a3-4fb4-b2c6-9440b3fa155b_isrForCrosstalkSources_998107_32.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 140 parsl.dataflow.strategy DEBUG: Executor multi has 140 active tasks, 1/0 running/pending blocks, and 0 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 34 more blocks parsl.executors.status_handling INFO: Scaling out by 34 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-15 22:46:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-15 22:46:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:46:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:46:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:46:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:46:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-15 22:46:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:46:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 8d9319e154abf5ad17172773306d83b7 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5963e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/d9bcb083-7c1b-4e96-98e1-70dac5f8fe42_isrForCrosstalkSources_998106_60.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/d9bcb083-7c1b-4e96-98e1-70dac5f8fe42_isrForCrosstalkSources_998106_60.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash 0be3ca8275a14829241f22654a0b26c2 parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a596340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/e61fdf29-faa3-40bf-aa75-6d7a373e2cfb_isrForCrosstalkSources_998107_3.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/e61fdf29-faa3-40bf-aa75-6d7a373e2cfb_isrForCrosstalkSources_998107_3.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 66af399809922b082c49189d72fa1266 parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5976a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/a6b66642-9faf-4597-a1ee-6ccc1f41e64c_isrForCrosstalkSources_998210_55.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/a6b66642-9faf-4597-a1ee-6ccc1f41e64c_isrForCrosstalkSources_998210_55.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash fbf5c47741c3f1331fcd1911468832e1 parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5977e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/172dafc9-b240-4038-8c39-f833947bb969_isrForCrosstalkSources_998211_10.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/172dafc9-b240-4038-8c39-f833947bb969_isrForCrosstalkSources_998211_10.stderr parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash c8a361ada80483d092954e7596bfa00c parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a597a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/15ce66a4-1af0-490b-a2dc-1c9aa8977ae3_isrForCrosstalkSources_998212_15.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/15ce66a4-1af0-490b-a2dc-1c9aa8977ae3_isrForCrosstalkSources_998212_15.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 54ab96973306751224eb171cf1efcba9 parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a597740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/21ca6938-da55-4469-814c-652e9d97a7f9_isrForCrosstalkSources_998108_4.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/21ca6938-da55-4469-814c-652e9d97a7f9_isrForCrosstalkSources_998108_4.stderr parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash c68ea77c1d13d1c44b002d6e25b3f707 parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a5979c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/39a34ee5-06a7-4257-b4cc-e580b3d65882_isrForCrosstalkSources_998106_59.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/39a34ee5-06a7-4257-b4cc-e580b3d65882_isrForCrosstalkSources_998106_59.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash c4bf3d2d92f04a1f893471897a223531 parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a597920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/7fe26f46-aa89-403d-95b3-e72fc287e147_isrForCrosstalkSources_998107_30.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/7fe26f46-aa89-403d-95b3-e72fc287e147_isrForCrosstalkSources_998107_30.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 8058bc5505dc0f20b954614e00cb7283 parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a597ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/bf016a6c-2280-45ab-bb21-0ea7c0815dcc_isrForCrosstalkSources_998211_47.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/bf016a6c-2280-45ab-bb21-0ea7c0815dcc_isrForCrosstalkSources_998211_47.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash 82540461e3639ee141ebcf1ce6f02034 parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a597b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/78b12a6b-df12-43cd-abc7-b0962b957ce2_isrForCrosstalkSources_998107_54.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/78b12a6b-df12-43cd-abc7-b0962b957ce2_isrForCrosstalkSources_998107_54.stderr parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash a39f823e9acce935e2c238fc104cb50e parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a597c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/e1e9cdda-ebb3-4d86-b627-5f1bf781af01_isrForCrosstalkSources_998211_22.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/e1e9cdda-ebb3-4d86-b627-5f1bf781af01_isrForCrosstalkSources_998211_22.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 3ce07465843da1bf00cca98857db9bf1 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a597e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/0a1441f9-6276-4b87-b0c1-45e7a5f79730_isrForCrosstalkSources_998106_8.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/0a1441f9-6276-4b87-b0c1-45e7a5f79730_isrForCrosstalkSources_998106_8.stderr parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash bb08c17b5326f3d78c9c2b2fdb551e37 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a597ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/2365cc45-d001-4dbb-a980-4a7999c47d8f_isrForCrosstalkSources_998106_22.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/2365cc45-d001-4dbb-a980-4a7999c47d8f_isrForCrosstalkSources_998106_22.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 26143a4ecf27cb9887945e9935087138 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a597d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/c20b6b04-0749-486f-8dcf-2ae4db78a7d0_isrForCrosstalkSources_998211_48.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/c20b6b04-0749-486f-8dcf-2ae4db78a7d0_isrForCrosstalkSources_998211_48.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash e1b0e99cd4455fc98e985899852d97a5 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a597f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/d711bc0a-b597-4e84-b8b1-f18ae0ee772a_isrForCrosstalkSources_998210_21.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/d711bc0a-b597-4e84-b8b1-f18ae0ee772a_isrForCrosstalkSources_998210_21.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 40c49403e1428eb29214845f0dd27f7e parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a597ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/b561cad7-1d7b-42c0-a40a-4ad9ca734561_isrForCrosstalkSources_998210_42.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/b561cad7-1d7b-42c0-a40a-4ad9ca734561_isrForCrosstalkSources_998210_42.stderr parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash de6fcf4abdafe36a93c6e37e1820b245 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a597880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/c4b5cd46-1c40-4e00-905d-fce322bdbb5f_isrForCrosstalkSources_998210_12.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/c4b5cd46-1c40-4e00-905d-fce322bdbb5f_isrForCrosstalkSources_998210_12.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash c4924b883ec8dd5d3b35d95243079752 parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/1681163f-aa34-45d8-b978-7b1b2c9e5cd8_isrForCrosstalkSources_998210_60.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/1681163f-aa34-45d8-b978-7b1b2c9e5cd8_isrForCrosstalkSources_998210_60.stderr parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 0d5fb12f7ac3e23125b99677bc41e9af parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/d28b95cc-94f4-4e3d-bf45-15f4965cb187_isrForCrosstalkSources_998212_13.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/d28b95cc-94f4-4e3d-bf45-15f4965cb187_isrForCrosstalkSources_998212_13.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 9194cfcaa1c519105b254b6fa4b42368 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/07649b61-4df9-4b4f-8c81-a08a82d7e934_isrForCrosstalkSources_998106_39.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/07649b61-4df9-4b4f-8c81-a08a82d7e934_isrForCrosstalkSources_998106_39.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash c6988287e6d25aecfe84d47372ec07cb parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/de7e65e4-2686-4760-8062-8d91d8587082_isrForCrosstalkSources_998210_20.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/de7e65e4-2686-4760-8062-8d91d8587082_isrForCrosstalkSources_998210_20.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash b7a5f8d81dea258c59edffb9e19671a7 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/25661b1e-64a5-4037-9f28-6652328af202_isrForCrosstalkSources_998108_22.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/25661b1e-64a5-4037-9f28-6652328af202_isrForCrosstalkSources_998108_22.stderr parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 4297fcfd7052c539090d0ce4acf9839b parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/21bbd208-b4ab-46d3-8556-6bca47ba0525_isrForCrosstalkSources_998108_32.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/21bbd208-b4ab-46d3-8556-6bca47ba0525_isrForCrosstalkSources_998108_32.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash e290f7a1f02dd040c7ca12bafb86941c parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/e170dc0a-2195-42dc-a4d5-c18283fb7cad_isrForCrosstalkSources_998211_33.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/e170dc0a-2195-42dc-a4d5-c18283fb7cad_isrForCrosstalkSources_998211_33.stderr parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash f215e744da1e14db06ab33dc2dcca6e5 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/895e3f75-99bd-4af1-80e5-8e57cf805422_isrForCrosstalkSources_998211_1.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/895e3f75-99bd-4af1-80e5-8e57cf805422_isrForCrosstalkSources_998211_1.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 7aa321a1d7133922108584569b72eff1 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/479bbbd9-c0da-498d-85bf-ac5b4c911753_isrForCrosstalkSources_998212_49.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/479bbbd9-c0da-498d-85bf-ac5b4c911753_isrForCrosstalkSources_998212_49.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 3b99ef7420885c0312318c53d9dae2fc parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/5942b0c1-9eb8-4a29-a390-b87cd92bd293_isrForCrosstalkSources_998106_37.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/5942b0c1-9eb8-4a29-a390-b87cd92bd293_isrForCrosstalkSources_998106_37.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 7dbfdb83170e5bcb515249678243c2a3 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/73a561e4-5d10-4835-81a0-006a563f7a2f_isrForCrosstalkSources_998107_53.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/73a561e4-5d10-4835-81a0-006a563f7a2f_isrForCrosstalkSources_998107_53.stderr parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash d0f7e762eddce492fb4f02df8d9861aa parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/c1c9d76d-66ca-454e-97cc-185bd2b45152_isrForCrosstalkSources_998108_14.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/c1c9d76d-66ca-454e-97cc-185bd2b45152_isrForCrosstalkSources_998108_14.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 3097e0224af802e2a25396e2632264ef parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/170d62af-2c83-44be-a1de-c845a449f5fc_isrForCrosstalkSources_998211_50.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/170d62af-2c83-44be-a1de-c845a449f5fc_isrForCrosstalkSources_998211_50.stderr parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 76480146e2c5363767aee2d67913f1ce parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/86639e85-a782-41d4-bbd3-1544aace3d34_isrForCrosstalkSources_998211_45.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/86639e85-a782-41d4-bbd3-1544aace3d34_isrForCrosstalkSources_998211_45.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash 5c1d4508eaa628de1238ba14b11c8c15 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/518e347f-eb09-4005-b46c-49f94399b3d8_isrForCrosstalkSources_998106_10.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/518e347f-eb09-4005-b46c-49f94399b3d8_isrForCrosstalkSources_998106_10.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash e60fba1b149cbb26ec4ecc342079d49a parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/8124448e-e27a-433c-8d9e-0323561a7a1c_isrForCrosstalkSources_998210_11.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/8124448e-e27a-433c-8d9e-0323561a7a1c_isrForCrosstalkSources_998210_11.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 39b7800ad14a7e7d909638dadd4ff36b parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/7bff279f-0038-49e7-a4e8-9da13cbd5ccd_isrForCrosstalkSources_998106_12.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/7bff279f-0038-49e7-a4e8-9da13cbd5ccd_isrForCrosstalkSources_998106_12.stderr parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash e233fc641ac19360f02fe5fddffb7db7 parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/c3a097f5-306a-4053-bc10-204ecbd9d95e_isrForCrosstalkSources_998210_5.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/c3a097f5-306a-4053-bc10-204ecbd9d95e_isrForCrosstalkSources_998210_5.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash dc5d6a3370bb3a8e513138544666f5cb parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/b713906a-5d3d-45a4-946e-eb1f23400edb_isrForCrosstalkSources_998210_26.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/b713906a-5d3d-45a4-946e-eb1f23400edb_isrForCrosstalkSources_998210_26.stderr parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 30350696ff776860b288200efa69f05e parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/5ada845f-f72a-4f12-b2c8-2ce6a6b1f387_isrForCrosstalkSources_998211_55.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/5ada845f-f72a-4f12-b2c8-2ce6a6b1f387_isrForCrosstalkSources_998211_55.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash f109bb5df13f168ed2fe910250aeba09 parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/57ca2f81-0663-4abb-b803-e57883758640_isrForCrosstalkSources_998210_13.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/57ca2f81-0663-4abb-b803-e57883758640_isrForCrosstalkSources_998210_13.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash d60eed9c5653a7d5c15a1b942b431665 parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/9034778d-c4d5-418e-9512-a15c6029dfa1_isrForCrosstalkSources_998211_3.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/9034778d-c4d5-418e-9512-a15c6029dfa1_isrForCrosstalkSources_998211_3.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 4c6586d67a95f4da98526e1904c13fd3 parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/b119e9a7-e2b9-4c51-aab3-094480255980_isrForCrosstalkSources_998108_46.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/b119e9a7-e2b9-4c51-aab3-094480255980_isrForCrosstalkSources_998108_46.stderr parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 3601cbf6f9fd6ef1442b12fbb1c168c2 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/63ae4365-280d-4ae4-a41f-7667a5d77d03_isrForCrosstalkSources_998212_5.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/63ae4365-280d-4ae4-a41f-7667a5d77d03_isrForCrosstalkSources_998212_5.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 911490187e713c017b88670301a79b5e parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/a6f91537-aa1c-4f79-85c4-c374c566804d_isrForCrosstalkSources_998211_4.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/a6f91537-aa1c-4f79-85c4-c374c566804d_isrForCrosstalkSources_998211_4.stderr parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash b8e5f2871f5e843321e6e72e2c2c43dc parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/ce553aff-6e38-4efc-b5cb-930d2281ec2f_isrForCrosstalkSources_998106_3.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/ce553aff-6e38-4efc-b5cb-930d2281ec2f_isrForCrosstalkSources_998106_3.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 4817503371a3194295fc13546a97483e parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/4f330e5d-0c22-489d-b60b-0bfc7d6f2f4a_isrForCrosstalkSources_998210_22.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/4f330e5d-0c22-489d-b60b-0bfc7d6f2f4a_isrForCrosstalkSources_998210_22.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash fc3abfe76a9d026e00b54c8c971289d2 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/d2071dec-4191-45a4-a12f-cfce8e0e3ef6_isrForCrosstalkSources_998108_45.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/d2071dec-4191-45a4-a12f-cfce8e0e3ef6_isrForCrosstalkSources_998108_45.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash e3946f2cf9562df671bfc1ec228b8872 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/1e299113-c1d9-47cb-ad9f-fb7c04a94dee_isrForCrosstalkSources_998211_27.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/1e299113-c1d9-47cb-ad9f-fb7c04a94dee_isrForCrosstalkSources_998211_27.stderr parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash d97ab5652539f142923157c3c536b8af parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/0bd62e64-126e-4e90-8c2c-c35e3e231fce_isrForCrosstalkSources_998106_52.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/0bd62e64-126e-4e90-8c2c-c35e3e231fce_isrForCrosstalkSources_998106_52.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash f7523579310ffb97bfb140d2a1bcdf03 parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/213b1071-e971-49d7-b4c4-f114e6b9cd91_isrForCrosstalkSources_998107_21.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/213b1071-e971-49d7-b4c4-f114e6b9cd91_isrForCrosstalkSources_998107_21.stderr parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 0d95c928b67040f1023349746ae20c1a parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/20f55edf-8725-4079-9e5b-120bbcf06886_isrForCrosstalkSources_998210_48.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/20f55edf-8725-4079-9e5b-120bbcf06886_isrForCrosstalkSources_998210_48.stderr parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash 30c79d88b75493fbbac52092bb1f726f parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/a02057c2-5bc7-411b-acc3-9a0d3da40359_isrForCrosstalkSources_998108_27.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/a02057c2-5bc7-411b-acc3-9a0d3da40359_isrForCrosstalkSources_998108_27.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash ac1bfb96d3d236cd8f786d4316ac85be parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/feb4ce61-dfad-4dda-988d-80f6d72aad51_isrForCrosstalkSources_998107_59.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/feb4ce61-dfad-4dda-988d-80f6d72aad51_isrForCrosstalkSources_998107_59.stderr parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 3b1daa90b01576835c988145affaea7b parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/422b3bcb-27db-445f-9d4f-673785ada959_isrForCrosstalkSources_998107_41.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/422b3bcb-27db-445f-9d4f-673785ada959_isrForCrosstalkSources_998107_41.stderr parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash cef1354faa93441945b7e2224606f484 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/3e4bf6ca-87ad-4248-872e-7fc54a878a1b_isrForCrosstalkSources_998108_23.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/3e4bf6ca-87ad-4248-872e-7fc54a878a1b_isrForCrosstalkSources_998108_23.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash bdc257a6332c4b026290b8911f9c2644 parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/5b4b7bd5-0d0f-4c58-9f7a-822da1205198_isrForCrosstalkSources_998212_59.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/5b4b7bd5-0d0f-4c58-9f7a-822da1205198_isrForCrosstalkSources_998212_59.stderr parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash 3e8858cab452f88e255bf3832e7ec138 parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/a6b459d3-90ed-4c7c-8a91-2a3c50217867_isrForCrosstalkSources_998108_39.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/a6b459d3-90ed-4c7c-8a91-2a3c50217867_isrForCrosstalkSources_998108_39.stderr parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 4b6aaa6923b3dc68c1ffc0d225eb1f0d parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/eb115207-35bf-444e-8809-650ac85bf1c7_isrForCrosstalkSources_998108_58.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/eb115207-35bf-444e-8809-650ac85bf1c7_isrForCrosstalkSources_998108_58.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash a4dfd713829d0861ce814e8fe1186515 parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/9c99b0f8-a763-419c-a1dc-3f63e7d3476d_isrForCrosstalkSources_998211_56.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/9c99b0f8-a763-419c-a1dc-3f63e7d3476d_isrForCrosstalkSources_998211_56.stderr parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash 719a72672bed669dfbacf6260fc98331 parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/e4d5d3d8-9ac0-44e3-91de-6b671d071d27_isrForCrosstalkSources_998108_35.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/e4d5d3d8-9ac0-44e3-91de-6b671d071d27_isrForCrosstalkSources_998108_35.stderr parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash e6020bc02eed38c04d7191755a3bd3bd parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/9137fde0-8da4-4957-b977-c6efce6b69a9_isrForCrosstalkSources_998107_57.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/9137fde0-8da4-4957-b977-c6efce6b69a9_isrForCrosstalkSources_998107_57.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash 4e2001725410051b5edba52eacff7216 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/5bc90cee-cf4c-4253-a72b-b30d79f3a9da_isrForCrosstalkSources_998210_1.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/5bc90cee-cf4c-4253-a72b-b30d79f3a9da_isrForCrosstalkSources_998210_1.stderr parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash adeec617eb96fcc89fa377c87034cbaa parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/d1126a1c-b361-4423-9efb-4775b153e34d_isrForCrosstalkSources_998108_20.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/d1126a1c-b361-4423-9efb-4775b153e34d_isrForCrosstalkSources_998108_20.stderr parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 377233d051343410ddced7d29ea1f07b parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/e5881988-e5fd-4096-beff-29eaf8a7dfb0_isrForCrosstalkSources_998106_21.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/e5881988-e5fd-4096-beff-29eaf8a7dfb0_isrForCrosstalkSources_998106_21.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 0ef53348d75e48588d9f48b0e510f6bd parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/9cc016b4-2bad-4678-b8ce-9ffbfda008b5_isrForCrosstalkSources_998210_17.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/9cc016b4-2bad-4678-b8ce-9ffbfda008b5_isrForCrosstalkSources_998210_17.stderr parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 144f7de07ce61dfa10f2ed9101085851 parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/7c64c40d-75f4-4301-a14e-fee326535477_isrForCrosstalkSources_998211_14.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/7c64c40d-75f4-4301-a14e-fee326535477_isrForCrosstalkSources_998211_14.stderr parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 30e11789c5bafce697249cd861091794 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/4f79728e-990f-4ef5-944c-6b06d57284e8_isrForCrosstalkSources_998211_61.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/4f79728e-990f-4ef5-944c-6b06d57284e8_isrForCrosstalkSources_998211_61.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 2ef21e28463a7f49a50ff084a21122a2 parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/d319d748-bffa-493e-bc93-117ada503282_isrForCrosstalkSources_998107_39.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/d319d748-bffa-493e-bc93-117ada503282_isrForCrosstalkSources_998107_39.stderr parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash a1895f8c9de17b68b5b22ec1be3b8b9d parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/96bc77a8-b49f-4059-bfe3-2ed5e38d19ba_isrForCrosstalkSources_998212_9.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/96bc77a8-b49f-4059-bfe3-2ed5e38d19ba_isrForCrosstalkSources_998212_9.stderr parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 3ba03212bb0003d6f662ce9e46c02252 parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/492b5709-e289-4eb7-b6fa-16095cf61ede_isrForCrosstalkSources_998211_31.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/492b5709-e289-4eb7-b6fa-16095cf61ede_isrForCrosstalkSources_998211_31.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 0edaad8eedd6b8f3bc8b26b1512a1ea6 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/2b662770-2dda-49b7-9218-b88617f9bee7_isrForCrosstalkSources_998212_3.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/2b662770-2dda-49b7-9218-b88617f9bee7_isrForCrosstalkSources_998212_3.stderr parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash ea3b2ac2171bb0d18ec1179244a11075 parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/5355d879-559d-4de5-871e-0ee0c68c1d6f_isrForCrosstalkSources_998106_42.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/5355d879-559d-4de5-871e-0ee0c68c1d6f_isrForCrosstalkSources_998106_42.stderr parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 9c0a70486c2d6d594e5696b1c9513965 parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/25aacc57-59f9-4c40-a28d-0fb3bb6c22a2_isrForCrosstalkSources_998210_46.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/25aacc57-59f9-4c40-a28d-0fb3bb6c22a2_isrForCrosstalkSources_998210_46.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash ea26eac2dff8cf8bd495bc770310363e parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 22:47:00 proc_lsst.multi:170 [INFO] [multi] job_id 17093085 proc_lsst.multi INFO: [multi] job_id 17093085 2024-03-15 22:47:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:00 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093085 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093085 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17093085 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-15 22:47:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-15 22:47:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/8bf4137a-efb1-48b8-bcc1-7e50536ffec8_isrForCrosstalkSources_998212_33.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/8bf4137a-efb1-48b8-bcc1-7e50536ffec8_isrForCrosstalkSources_998212_33.stderr parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash cb3d31447f3bce2af8876f48c632bae4 parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a596520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/c5e20072-3d0d-425f-82db-da8c6d93d183_isrForCrosstalkSources_998107_25.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/c5e20072-3d0d-425f-82db-da8c6d93d183_isrForCrosstalkSources_998107_25.stderr parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash c8e2a4e87b2b5e1ed76bfa29a89c429c parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a594040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/213d8a3e-f108-4f27-866d-eeba41df7b70_isrForCrosstalkSources_998210_15.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/213d8a3e-f108-4f27-866d-eeba41df7b70_isrForCrosstalkSources_998210_15.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash 594084c492848c9cb59b595b6297b0ff parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a597600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/6d4f0b1d-fdba-4292-a0dc-7e9c2103f805_isrForCrosstalkSources_998212_27.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/6d4f0b1d-fdba-4292-a0dc-7e9c2103f805_isrForCrosstalkSources_998212_27.stderr parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash 8c59732b676a653f7fc00326658b39e4 parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679025760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/2a2c0d71-4635-406c-989f-a5e0c928994c_isrForCrosstalkSources_998107_40.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/2a2c0d71-4635-406c-989f-a5e0c928994c_isrForCrosstalkSources_998107_40.stderr parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash bd91e699b785a0c65d3cf1b7a6fd4a4e parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/20631956-4fdb-470c-a834-776015970c50_isrForCrosstalkSources_998107_5.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/20631956-4fdb-470c-a834-776015970c50_isrForCrosstalkSources_998107_5.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 9e7494d6ff09eadca97f33e6b27dd245 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/990a6e21-5e7b-4e73-9d1c-2211a8b628fb_isrForCrosstalkSources_998212_24.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/990a6e21-5e7b-4e73-9d1c-2211a8b628fb_isrForCrosstalkSources_998212_24.stderr parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 8b82a67eae199142b9bda4783afc7af3 parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/a6f14c72-be9c-4902-8087-9134bfe68482_isrForCrosstalkSources_998106_43.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/a6f14c72-be9c-4902-8087-9134bfe68482_isrForCrosstalkSources_998106_43.stderr parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash de1b62772cb6deb78a2881676804e8b8 parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/9ef6f49a-b34b-4b70-9774-e79b4d68c914_isrForCrosstalkSources_998212_36.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/9ef6f49a-b34b-4b70-9774-e79b4d68c914_isrForCrosstalkSources_998212_36.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash dd0297266c93f64f19ebbcd8f09bdcf2 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/a36475b7-b60a-4540-af70-88012920a296_isrForCrosstalkSources_998211_15.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/a36475b7-b60a-4540-af70-88012920a296_isrForCrosstalkSources_998211_15.stderr parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash 517984e43719ae2bde89a439f874d87b parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/f0c786b9-e7a0-4940-bf74-aad384c05a83_isrForCrosstalkSources_998211_19.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/f0c786b9-e7a0-4940-bf74-aad384c05a83_isrForCrosstalkSources_998211_19.stderr parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash 41790b03ebed2cfe937a2fa7f7425b48 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/24fa03c4-a4b0-4a19-9c8b-bb0729063bf0_isrForCrosstalkSources_998212_26.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/24fa03c4-a4b0-4a19-9c8b-bb0729063bf0_isrForCrosstalkSources_998212_26.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 145c8cfe13f217a3c0725aa765e95393 parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/e2447ac6-eebb-4658-ae58-abb7df9ca1cf_isrForCrosstalkSources_998108_61.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/e2447ac6-eebb-4658-ae58-abb7df9ca1cf_isrForCrosstalkSources_998108_61.stderr parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash f6d1e417b8ddf4323d308e6d0099ce7c parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/79efce15-b5d5-4237-8ed5-2addb197daf6_isrForCrosstalkSources_998106_44.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/79efce15-b5d5-4237-8ed5-2addb197daf6_isrForCrosstalkSources_998106_44.stderr parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash bb09926ab98bd3eea98f013cd28adf8b parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/26a4e535-99a5-4433-bfb4-e01fe9a67b29_isrForCrosstalkSources_998211_58.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/26a4e535-99a5-4433-bfb4-e01fe9a67b29_isrForCrosstalkSources_998211_58.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash 5cc79fdbdcc8bf5a17419d9dad80a90c parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/76a355a1-b4f8-4bc5-8091-cd7bf4326688_isrForCrosstalkSources_998106_36.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/76a355a1-b4f8-4bc5-8091-cd7bf4326688_isrForCrosstalkSources_998106_36.stderr parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash a86de2052381e1b54d3d8d43732f2a13 parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/62e08bb1-ba2d-45a5-86af-ca5012bdf78f_isrForCrosstalkSources_998108_44.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/62e08bb1-ba2d-45a5-86af-ca5012bdf78f_isrForCrosstalkSources_998108_44.stderr parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash e8b7697ec2698d7d6b8e147aba6b9f99 parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/4cca31ce-c18e-4630-8472-f79147039b46_isrForCrosstalkSources_998106_19.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/4cca31ce-c18e-4630-8472-f79147039b46_isrForCrosstalkSources_998106_19.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash 8c622acd63c95ea1b69a71259ca3af28 parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/e2200ead-d0ab-422c-bd64-af6b09305341_isrForCrosstalkSources_998108_53.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/e2200ead-d0ab-422c-bd64-af6b09305341_isrForCrosstalkSources_998108_53.stderr parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash f79b5ae2d1e57d219eda74f9e1d8ac22 parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/6b02e212-4433-4246-8a83-c79988117291_isrForCrosstalkSources_998106_47.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/6b02e212-4433-4246-8a83-c79988117291_isrForCrosstalkSources_998106_47.stderr parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash fa3bfc5f040f0af4dc79c722d3fa68e2 parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/5a0aa601-a5c8-4f51-8830-943a6eac3a44_isrForCrosstalkSources_998106_14.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/5a0aa601-a5c8-4f51-8830-943a6eac3a44_isrForCrosstalkSources_998106_14.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash e9270438be8272ed2847cf8eddad55e1 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/1bea362c-d4be-4735-b0eb-1ae26d040b38_isrForCrosstalkSources_998211_39.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/1bea362c-d4be-4735-b0eb-1ae26d040b38_isrForCrosstalkSources_998211_39.stderr parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash 291289a09898b2323b2e88c4272d597c parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/16799f2f-914a-466b-9db0-5ffe844b798d_isrForCrosstalkSources_998106_49.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/16799f2f-914a-466b-9db0-5ffe844b798d_isrForCrosstalkSources_998106_49.stderr parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 245e5719fa2db2077def113703b817c6 parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/edf15cac-99ba-4162-8789-930968fd49a5_isrForCrosstalkSources_998108_8.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/edf15cac-99ba-4162-8789-930968fd49a5_isrForCrosstalkSources_998108_8.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash a56a9ead4f397a5b42d21cae6dba904f parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/06e4b068-c4f2-412c-95f8-7d3ff2f81b8f_isrForCrosstalkSources_998107_61.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/06e4b068-c4f2-412c-95f8-7d3ff2f81b8f_isrForCrosstalkSources_998107_61.stderr parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash 94dcf2bd64313d5ed09932a5e8c1b7de parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/ad3613c1-79f4-4ef5-b05a-e1f45ab0b522_isrForCrosstalkSources_998107_46.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/ad3613c1-79f4-4ef5-b05a-e1f45ab0b522_isrForCrosstalkSources_998107_46.stderr parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 9dc9d7828cfc2d72aa91fd1abd09e759 parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/de44599d-a8ac-4b39-bc45-b59cc6661aff_isrForCrosstalkSources_998107_7.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/de44599d-a8ac-4b39-bc45-b59cc6661aff_isrForCrosstalkSources_998107_7.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 6196e311ea6bb0b62754cefb655a8541 parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/4f34e4f8-d881-4628-ba9f-cae7892d0bae_isrForCrosstalkSources_998107_9.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/4f34e4f8-d881-4628-ba9f-cae7892d0bae_isrForCrosstalkSources_998107_9.stderr parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash 8920fdc0b488c70bc97856d7f8a2e3ad parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/24280292-ae6f-4eff-81d7-a22e198c6511_isrForCrosstalkSources_998212_19.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/24280292-ae6f-4eff-81d7-a22e198c6511_isrForCrosstalkSources_998212_19.stderr parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 0704c7e00234bf039932afd9ee8c316b parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/403f5852-4d9c-402b-a024-60a4375dabae_isrForCrosstalkSources_998107_51.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/403f5852-4d9c-402b-a024-60a4375dabae_isrForCrosstalkSources_998107_51.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 5216cc2fa1b7a437ffe1252d88b7d433 parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/00b01b71-2a84-49c7-ac59-4e8f9a99dee0_isrForCrosstalkSources_998108_49.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/00b01b71-2a84-49c7-ac59-4e8f9a99dee0_isrForCrosstalkSources_998108_49.stderr parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 8a1c17547fe921fd0d7c135162aa6105 parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/2af379ee-c027-407e-9bc5-587c3918412a_isrForCrosstalkSources_998212_6.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/2af379ee-c027-407e-9bc5-587c3918412a_isrForCrosstalkSources_998212_6.stderr parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash 0d0aa7cd190077096229dc30dfdfdaae parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/2cc66ffa-12cc-4800-b60b-c82259c2651a_isrForCrosstalkSources_998107_4.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/2cc66ffa-12cc-4800-b60b-c82259c2651a_isrForCrosstalkSources_998107_4.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 575edec2a6f6fdecc71f0066b0537759 parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/06738163-c244-4c9e-a7ec-709f962ce037_isrForCrosstalkSources_998106_34.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/06738163-c244-4c9e-a7ec-709f962ce037_isrForCrosstalkSources_998106_34.stderr parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash c5fc3327a1ee7e07c8908ef8f4b3789e parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/7cf048ed-549d-4a3b-9092-826625ed733c_isrForCrosstalkSources_998106_38.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/7cf048ed-549d-4a3b-9092-826625ed733c_isrForCrosstalkSources_998106_38.stderr parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash dccb0d47c7b305ad3191e26221263701 parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/278cdfb2-7dd5-494d-809b-6feed3a45aca_isrForCrosstalkSources_998212_53.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/278cdfb2-7dd5-494d-809b-6feed3a45aca_isrForCrosstalkSources_998212_53.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash f1ccb948a6fdb1a4ea40902b7ff89a06 parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/909dcc40-5375-42c6-b21a-36bdcd6f1ef0_isrForCrosstalkSources_998106_50.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/909dcc40-5375-42c6-b21a-36bdcd6f1ef0_isrForCrosstalkSources_998106_50.stderr parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash ace8be516d54cf75507aa611c7241a07 parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 22:47:01 proc_lsst.multi:170 [INFO] [multi] job_id 17093086 proc_lsst.multi INFO: [multi] job_id 17093086 2024-03-15 22:47:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 22:47:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:01 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093086 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093086 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17093086 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-15 22:47:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-15 22:47:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 22:47:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/6f7efc4b-7738-4c41-90fa-a8cbfb9ce32a_isrForCrosstalkSources_998210_23.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/6f7efc4b-7738-4c41-90fa-a8cbfb9ce32a_isrForCrosstalkSources_998210_23.stderr parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash abec207e69eab9f30b2274e1cfecef9d parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/acf263cb-18dc-4b19-b01d-438f5617d45c_isrForCrosstalkSources_998107_47.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/acf263cb-18dc-4b19-b01d-438f5617d45c_isrForCrosstalkSources_998107_47.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash 69fee44679029a98fd395f7bc57e2d2f parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/9f519d04-1eea-489b-beab-e05730b6685a_isrForCrosstalkSources_998212_50.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/9f519d04-1eea-489b-beab-e05730b6685a_isrForCrosstalkSources_998212_50.stderr parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash 1cf1ce25a474689932451f973db5b8bf parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/1637a91c-ea80-43dd-bb8a-4bab0abcce74_isrForCrosstalkSources_998211_34.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/1637a91c-ea80-43dd-bb8a-4bab0abcce74_isrForCrosstalkSources_998211_34.stderr parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash 39db09ba7fc456375abcc685bf259aa6 parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b67a597560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/500d6c2d-b30c-48fd-bf9e-dd5f5a3c9571_isrForCrosstalkSources_998210_14.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/500d6c2d-b30c-48fd-bf9e-dd5f5a3c9571_isrForCrosstalkSources_998210_14.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 9586df0c7ce533610c722c29ba0e8355 parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/304ad50c-49f7-4573-a0f0-f6d3cab741cb_isrForCrosstalkSources_998108_36.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/304ad50c-49f7-4573-a0f0-f6d3cab741cb_isrForCrosstalkSources_998108_36.stderr parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash 55afd4fde77bc28babcfb7325d53dd11 parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679027060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/df8bf773-b056-4ad5-a248-43c10d0b0e9b_isrForCrosstalkSources_998108_41.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/df8bf773-b056-4ad5-a248-43c10d0b0e9b_isrForCrosstalkSources_998108_41.stderr parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash d3311df3deff562241362294b78c484c parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/8ad5a76f-6f21-47cc-bdab-1fb4e7d1a6f8_isrForCrosstalkSources_998211_51.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/8ad5a76f-6f21-47cc-bdab-1fb4e7d1a6f8_isrForCrosstalkSources_998211_51.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash 5c6ca9625b598bf42dd4960effe10be9 parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/c5a4870e-43ae-4d48-aae7-ee572e416317_isrForCrosstalkSources_998106_57.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/c5a4870e-43ae-4d48-aae7-ee572e416317_isrForCrosstalkSources_998106_57.stderr parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash 564c3cccc615242958d80d6312c72359 parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/21c29fd7-ea1f-4b73-872b-07edf107e9f3_isrForCrosstalkSources_998107_45.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/21c29fd7-ea1f-4b73-872b-07edf107e9f3_isrForCrosstalkSources_998107_45.stderr parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash 746e5eeefb37992f5eb23cb7fc5c2596 parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/ebb28cfa-7e9c-4a00-9a73-ed9844c9955f_isrForCrosstalkSources_998211_8.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/ebb28cfa-7e9c-4a00-9a73-ed9844c9955f_isrForCrosstalkSources_998211_8.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 4595ef186a48114fa906978091bfe644 parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/d82624fa-fe3a-4ef6-8ee2-32a3e14585b5_isrForCrosstalkSources_998211_49.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/d82624fa-fe3a-4ef6-8ee2-32a3e14585b5_isrForCrosstalkSources_998211_49.stderr parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash 89816acc6f766db3e5fa3fa41cb6c035 parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/17cdcac1-a0f0-41dc-81b4-aaf1ae4546bb_isrForCrosstalkSources_998212_42.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/17cdcac1-a0f0-41dc-81b4-aaf1ae4546bb_isrForCrosstalkSources_998212_42.stderr parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash c15a93342bf120a251b4fb93509746c9 parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679026fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/012f14bd-61e6-4d46-976e-f8ecbbc51b74_isrForCrosstalkSources_998108_28.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/012f14bd-61e6-4d46-976e-f8ecbbc51b74_isrForCrosstalkSources_998108_28.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash 274326c1ad07c8abc14995c5b217b0bf parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679025580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/162bf8db-74a7-4a17-9e5f-750254ec44fa_isrForCrosstalkSources_998106_23.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/162bf8db-74a7-4a17-9e5f-750254ec44fa_isrForCrosstalkSources_998106_23.stderr parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash 546da228fe4b74c533186a2bedab2fa6 parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679024040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/b390a8b5-1cd5-457b-8a49-432c08e811b6_isrForCrosstalkSources_998107_27.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/b390a8b5-1cd5-457b-8a49-432c08e811b6_isrForCrosstalkSources_998107_27.stderr parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash 71d9b944fd1ec45261b5f535b4da2712 parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790256c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/297620a5-e7c1-44e2-b3fb-ef01eba9eea7_isrForCrosstalkSources_998106_45.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/297620a5-e7c1-44e2-b3fb-ef01eba9eea7_isrForCrosstalkSources_998106_45.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 82069a7126edfc7b323e74b180e0a442 parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679024180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/4af724b7-2872-4c3a-bb4a-08bfa4a693e4_isrForCrosstalkSources_998212_54.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/4af724b7-2872-4c3a-bb4a-08bfa4a693e4_isrForCrosstalkSources_998212_54.stderr parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash 363cdbc3596adb930f83e5cf030e6436 parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679024220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/322552ad-71f6-4cfb-83c4-ebae04cf35e4_isrForCrosstalkSources_998106_24.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/322552ad-71f6-4cfb-83c4-ebae04cf35e4_isrForCrosstalkSources_998106_24.stderr parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash 183349d73451a4c4eb800ab8d4725d8e parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790240e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/82a4c72a-ee6b-4401-9b11-f20c7738fda7_isrForCrosstalkSources_998211_43.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/82a4c72a-ee6b-4401-9b11-f20c7738fda7_isrForCrosstalkSources_998211_43.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash b1bad638b957592d01556848fa71d811 parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679024400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/ebd46c56-d350-4f28-a644-247d6c6e245d_isrForCrosstalkSources_998106_62.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/ebd46c56-d350-4f28-a644-247d6c6e245d_isrForCrosstalkSources_998106_62.stderr parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash 7a6b998c641bb82a37398a3143b81eab parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790242c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/ab0b52a3-6850-49de-9952-5c75977f3ebf_isrForCrosstalkSources_998210_52.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/ab0b52a3-6850-49de-9952-5c75977f3ebf_isrForCrosstalkSources_998210_52.stderr parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash be720451a21e621c5102827fb951b349 parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679025800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/a3cb58f3-47a7-4072-84f4-34ffd616d227_isrForCrosstalkSources_998210_58.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/a3cb58f3-47a7-4072-84f4-34ffd616d227_isrForCrosstalkSources_998210_58.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash d699585e44393aab133886f5df962d61 parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790254e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/63b68206-41b2-4396-a788-92ea1fe72118_isrForCrosstalkSources_998210_54.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/63b68206-41b2-4396-a788-92ea1fe72118_isrForCrosstalkSources_998210_54.stderr parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash 3755fe179546b8b3e8ca84084af32eea parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679025440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/45272384-07c5-4430-a32c-d5099ea60c52_isrForCrosstalkSources_998211_11.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/45272384-07c5-4430-a32c-d5099ea60c52_isrForCrosstalkSources_998211_11.stderr parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash d3b05b7e3e6a9cb7943d2ac0fe6a4066 parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679025620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/0faa5d51-d329-4411-b22c-a1d45ad2055c_isrForCrosstalkSources_998106_9.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/0faa5d51-d329-4411-b22c-a1d45ad2055c_isrForCrosstalkSources_998106_9.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 421a31c84a7737e8547a45bf9c9424ad parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790253a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/0b2b9cc6-77e9-4d96-a637-de4eb7c6a139_isrForCrosstalkSources_998211_53.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/0b2b9cc6-77e9-4d96-a637-de4eb7c6a139_isrForCrosstalkSources_998211_53.stderr parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash b971ab94597f3191e077737ae930660a parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679025300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/a7cfbdce-2569-410d-902e-47452775fd63_isrForCrosstalkSources_998107_36.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/a7cfbdce-2569-410d-902e-47452775fd63_isrForCrosstalkSources_998107_36.stderr parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash 36141c2a6ed5c560522f211394cb86a7 parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790251c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/bc65387a-3ef0-4172-b282-4b28221eba3d_isrForCrosstalkSources_998108_55.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/bc65387a-3ef0-4172-b282-4b28221eba3d_isrForCrosstalkSources_998108_55.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash b394fec2e7f45ca31b072e7791b35ca3 parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679025120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/0761ff90-1dad-487f-a850-ae2e01d2a203_isrForCrosstalkSources_998210_35.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/0761ff90-1dad-487f-a850-ae2e01d2a203_isrForCrosstalkSources_998210_35.stderr parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 2c935a420f2381fafb25ee8c8e40f432 parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679025260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/93061d01-4afd-4e7c-8929-7e5ca6f9c9fe_isrForCrosstalkSources_998212_55.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/93061d01-4afd-4e7c-8929-7e5ca6f9c9fe_isrForCrosstalkSources_998212_55.stderr parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash da95e01234c6da9416d2b3ab82a09262 parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679025080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/6a394dc5-9472-46a0-afbf-168e62da50e8_isrForCrosstalkSources_998106_53.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/6a394dc5-9472-46a0-afbf-168e62da50e8_isrForCrosstalkSources_998106_53.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash a290513a2d2c21b2016b4435f3fa683a parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679024f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/efe8389e-abe5-454f-8dbd-969adf99b352_isrForCrosstalkSources_998108_62.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/efe8389e-abe5-454f-8dbd-969adf99b352_isrForCrosstalkSources_998108_62.stderr parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash e0e1e93e65e4ae655a9118275db799ba parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679024fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/1501d4e1-8a40-461e-a521-a4cc1b994522_isrForCrosstalkSources_998212_10.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/1501d4e1-8a40-461e-a521-a4cc1b994522_isrForCrosstalkSources_998212_10.stderr parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash 79957d607befd1363edc6d7cf824f13b parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679024ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/bfcd9c85-cfd6-459b-9961-f1722df5f117_isrForCrosstalkSources_998212_20.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/bfcd9c85-cfd6-459b-9961-f1722df5f117_isrForCrosstalkSources_998212_20.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash be8fa8089709ecb010f58965d93d0ec2 parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679024e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/d06c7bbe-5f50-49fd-bf9f-170109447e02_isrForCrosstalkSources_998210_2.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/d06c7bbe-5f50-49fd-bf9f-170109447e02_isrForCrosstalkSources_998210_2.stderr parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash 66c330fc02db77267aa8dd9bca580847 parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679024d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 22:47:01 proc_lsst.multi:170 [INFO] [multi] job_id 17093087 proc_lsst.multi INFO: [multi] job_id 17093087 2024-03-15 22:47:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 22:47:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:01 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093087 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093087 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17093087 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-15 22:47:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-15 22:47:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 22:47:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/3196228e-e312-441d-aa56-04a9f6c68d30_isrForCrosstalkSources_998108_18.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/3196228e-e312-441d-aa56-04a9f6c68d30_isrForCrosstalkSources_998108_18.stderr parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash 4e3f69f4572359745bdcbe4bd8b5f892 parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/a7d8996b-3863-47b5-97d1-2764cfe42064_isrForCrosstalkSources_998212_34.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/a7d8996b-3863-47b5-97d1-2764cfe42064_isrForCrosstalkSources_998212_34.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash c60b5610b4f885d0c353ab27c5a3133f parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/4cdecb8d-7f37-4b8b-9493-177541b981cd_isrForCrosstalkSources_998106_15.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/4cdecb8d-7f37-4b8b-9493-177541b981cd_isrForCrosstalkSources_998106_15.stderr parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 6c0e87fc636db19b6e4278d5a92b9a30 parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679026de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/2ad18b02-2107-4c23-a37e-48f9d3adf516_isrForCrosstalkSources_998106_32.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/2ad18b02-2107-4c23-a37e-48f9d3adf516_isrForCrosstalkSources_998106_32.stderr parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash e0bd3025765139e6b47083e44ca9b2de parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678fc3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/d20cc9d1-5108-462c-9564-e0c3327e44a3_isrForCrosstalkSources_998212_41.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/d20cc9d1-5108-462c-9564-e0c3327e44a3_isrForCrosstalkSources_998212_41.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 7ceeeeb5f27dfbbdc4749d766a9f3a96 parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679024b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/cf296b05-9b6e-4dbd-928d-20f1e139f820_isrForCrosstalkSources_998107_60.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/cf296b05-9b6e-4dbd-928d-20f1e139f820_isrForCrosstalkSources_998107_60.stderr parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash b854e00178c0ff7837550089e6ec5c21 parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679027e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/842dfabd-60b1-4e89-9d56-868b60d489d5_isrForCrosstalkSources_998212_7.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/842dfabd-60b1-4e89-9d56-868b60d489d5_isrForCrosstalkSources_998212_7.stderr parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash e8e903aa1a991edd0cff2118bb58b47b parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679024c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/e39de0c9-7b9a-4ee2-8cca-67ae3d533706_isrForCrosstalkSources_998107_52.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/e39de0c9-7b9a-4ee2-8cca-67ae3d533706_isrForCrosstalkSources_998107_52.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash 2b384b2082194b4dc47a7e1dec9dfbae parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679027ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/0f27bc96-0c0d-4011-a836-3e4e6d5db5c3_isrForCrosstalkSources_998108_43.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/0f27bc96-0c0d-4011-a836-3e4e6d5db5c3_isrForCrosstalkSources_998108_43.stderr parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash 6f97d222f331a9daaca76e92e242eb43 parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679027ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/f8c3ab26-add9-4189-8e67-f5055757a1ce_isrForCrosstalkSources_998210_39.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/f8c3ab26-add9-4189-8e67-f5055757a1ce_isrForCrosstalkSources_998210_39.stderr parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash 7c41f6b43aefd09b855e2566b2542a62 parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679027d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/3c5e8f7e-0e29-4d06-8beb-026269fe6e1b_isrForCrosstalkSources_998210_16.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/3c5e8f7e-0e29-4d06-8beb-026269fe6e1b_isrForCrosstalkSources_998210_16.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash a079fa0c323cfdb60b2079f9eb75b913 parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679024900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/62c737e3-f590-429c-bbca-074257cf2176_isrForCrosstalkSources_998212_38.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/62c737e3-f590-429c-bbca-074257cf2176_isrForCrosstalkSources_998212_38.stderr parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 5162f6ba37785000559017df2dcb9691 parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679027ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/979b44f1-56a9-4ab5-b0fc-da90c089f8a4_isrForCrosstalkSources_998106_13.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/979b44f1-56a9-4ab5-b0fc-da90c089f8a4_isrForCrosstalkSources_998106_13.stderr parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash f167f8cbe1f5912f3e6ecc5ef68bc40c parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679024720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/b639afbd-0505-4646-8156-b2900a96cc19_isrForCrosstalkSources_998106_48.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/b639afbd-0505-4646-8156-b2900a96cc19_isrForCrosstalkSources_998106_48.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash 43ccdec4feac7e800c74b537dc9f4954 parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790245e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/f644d4da-d287-48c5-9780-d2b770cb9015_isrForCrosstalkSources_998210_27.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/f644d4da-d287-48c5-9780-d2b770cb9015_isrForCrosstalkSources_998210_27.stderr parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash 281dacab8f3fad924b4569a115ecc276 parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679024860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/e96743cd-e6db-4605-8cf9-95516325461b_isrForCrosstalkSources_998210_49.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/e96743cd-e6db-4605-8cf9-95516325461b_isrForCrosstalkSources_998210_49.stderr parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash a9f931b95ef0f3b41cc619bd3c6c3e0f parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790247c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/f56d6149-b396-4cac-8fef-1ca55daa58e2_isrForCrosstalkSources_998212_12.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/f56d6149-b396-4cac-8fef-1ca55daa58e2_isrForCrosstalkSources_998212_12.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash 14b4b213f773b40b325d0dbce77853aa parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679024540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/9fb06955-ed8d-419a-ac52-e1152ac98988_isrForCrosstalkSources_998106_20.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/9fb06955-ed8d-419a-ac52-e1152ac98988_isrForCrosstalkSources_998106_20.stderr parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash c6bd5acdb21c075adf1a5373f732479e parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790244a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/ac1495fc-8a50-4e61-9204-87010421cb4f_isrForCrosstalkSources_998107_16.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/ac1495fc-8a50-4e61-9204-87010421cb4f_isrForCrosstalkSources_998107_16.stderr parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash a69475ba70b5011f7187216b430c6cad parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679025940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/89b45a64-8c3e-45b4-8757-6fc37866b8c3_isrForCrosstalkSources_998212_62.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/89b45a64-8c3e-45b4-8757-6fc37866b8c3_isrForCrosstalkSources_998212_62.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash 845427ac8b7e7835b33b79ebae4f977f parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679024680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/dcd8f7a8-25c1-4881-a301-6f1b92410a46_isrForCrosstalkSources_998107_6.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/dcd8f7a8-25c1-4881-a301-6f1b92410a46_isrForCrosstalkSources_998107_6.stderr parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash e48f998c02b2c32326c613a728c0811d parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790259e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/ccf6d565-cc17-45ca-89d8-0efddadf3004_isrForCrosstalkSources_998211_7.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/ccf6d565-cc17-45ca-89d8-0efddadf3004_isrForCrosstalkSources_998211_7.stderr parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash bbee717e28bbe6f05ae25df30e050f3b parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790258a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/cbc042b2-10b4-4562-bc40-003ceaa1b343_isrForCrosstalkSources_998106_26.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/cbc042b2-10b4-4562-bc40-003ceaa1b343_isrForCrosstalkSources_998106_26.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash 33b3b6611450ee961bc06c2779be672f parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679024360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/1d8347c1-0c1f-43f6-bf66-169926410ed0_isrForCrosstalkSources_998212_30.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/1d8347c1-0c1f-43f6-bf66-169926410ed0_isrForCrosstalkSources_998212_30.stderr parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash 4a8ce5be83637bacc65855adab52eb68 parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679025b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/521b04b5-c249-4e72-888d-5388fb1302f0_isrForCrosstalkSources_998107_13.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/521b04b5-c249-4e72-888d-5388fb1302f0_isrForCrosstalkSources_998107_13.stderr parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 3175fa531561eafc1c13763f1561dde7 parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679025a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/633ec7d0-b2a3-4738-9214-b9feaa94d4be_isrForCrosstalkSources_998210_7.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/633ec7d0-b2a3-4738-9214-b9feaa94d4be_isrForCrosstalkSources_998210_7.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash db2177afbdc506f56b0ed7379c8aa947 parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679026f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/0ef607d7-b319-46d4-b84c-f8d4957f37f4_isrForCrosstalkSources_998106_41.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/0ef607d7-b319-46d4-b84c-f8d4957f37f4_isrForCrosstalkSources_998106_41.stderr parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash 2e1b9c66cba3ff32c64cb1e6dfa4b9de parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679026e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/d912e4b2-4bf4-433c-b51e-1458aaf8c489_isrForCrosstalkSources_998211_38.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/d912e4b2-4bf4-433c-b51e-1458aaf8c489_isrForCrosstalkSources_998211_38.stderr parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash ae458580b2f370df1dd114594a033156 parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679026d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/2d38db2e-3ce8-4541-8b55-5deb711ed48e_isrForCrosstalkSources_998211_52.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/2d38db2e-3ce8-4541-8b55-5deb711ed48e_isrForCrosstalkSources_998211_52.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash 5b6fd9fe84a9cec896b7c95d1b6f6384 parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679026ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/523c94db-3532-47f4-b4e3-f241fb6ebef8_isrForCrosstalkSources_998210_38.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/523c94db-3532-47f4-b4e3-f241fb6ebef8_isrForCrosstalkSources_998210_38.stderr parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash 9afdc1210b8412161f16ff1c19230cb7 parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679025c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/999e6821-67e8-4c6f-bbbc-89d5adbe6b39_isrForCrosstalkSources_998107_2.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/999e6821-67e8-4c6f-bbbc-89d5adbe6b39_isrForCrosstalkSources_998107_2.stderr parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash 013b21366468ae61f6d576a987d776e7 parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679026b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/3e4db861-1aab-46a9-a69a-fc92273b4982_isrForCrosstalkSources_998106_61.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/3e4db861-1aab-46a9-a69a-fc92273b4982_isrForCrosstalkSources_998106_61.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 02adb19e87e028cabfe94635e5dd436d parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679026ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/435ca4ad-528c-438e-b1f4-f0c915b8c25e_isrForCrosstalkSources_998106_31.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/435ca4ad-528c-438e-b1f4-f0c915b8c25e_isrForCrosstalkSources_998106_31.stderr parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash 5803f8bfff1bd02cdf7fa01520403b10 parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679026a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/6747991f-5486-4de1-8a7e-4bf776291fc2_isrForCrosstalkSources_998211_2.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/6747991f-5486-4de1-8a7e-4bf776291fc2_isrForCrosstalkSources_998211_2.stderr parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash 7e4e63ade00d77a7713b3281850af0a4 parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790268e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/8e4e33f6-2c1e-46e7-a7e3-311ebc4ba78e_isrForCrosstalkSources_998107_62.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/8e4e33f6-2c1e-46e7-a7e3-311ebc4ba78e_isrForCrosstalkSources_998107_62.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash dcbebf280950518bd19eaf562ed03554 parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679026c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/db6c609c-8098-40ce-b917-5df55238f842_isrForCrosstalkSources_998212_56.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/db6c609c-8098-40ce-b917-5df55238f842_isrForCrosstalkSources_998212_56.stderr parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash 0884b543bcf068dfd151fac65da11f6d parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679026980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/81da7d2b-c892-4da0-886b-3510d099e1cc_isrForCrosstalkSources_998107_28.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/81da7d2b-c892-4da0-886b-3510d099e1cc_isrForCrosstalkSources_998107_28.stderr parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash 5ca4c805cbbacd8629f8f9deda932c3e parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679026840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/337a5853-d946-431d-8359-80ab18118dfb_isrForCrosstalkSources_998107_35.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/337a5853-d946-431d-8359-80ab18118dfb_isrForCrosstalkSources_998107_35.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash e52d96a6076258473fa3f961ea72b41f parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679026700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/608856b2-cac9-4c09-a7cc-ea6eb2728e8f_isrForCrosstalkSources_998107_26.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/608856b2-cac9-4c09-a7cc-ea6eb2728e8f_isrForCrosstalkSources_998107_26.stderr parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 2cc66ee8f31130d16f9ae4740039b6c0 parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790265c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/29bdd4a9-1a7b-492f-87fa-2e9f18e6cd75_isrForCrosstalkSources_998212_45.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/29bdd4a9-1a7b-492f-87fa-2e9f18e6cd75_isrForCrosstalkSources_998212_45.stderr parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash 63621e08e54d3cd8fa822be4a902f068 parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790267a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/eb0f1aa6-d52b-40d7-9b48-c8b7b2e3780c_isrForCrosstalkSources_998107_17.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/eb0f1aa6-d52b-40d7-9b48-c8b7b2e3780c_isrForCrosstalkSources_998107_17.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 2e9d82ec09c686e903155d1e5618d650 parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679026660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/064ea1fa-08b9-48b3-a09c-54921648e171_isrForCrosstalkSources_998210_8.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/064ea1fa-08b9-48b3-a09c-54921648e171_isrForCrosstalkSources_998210_8.stderr parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash d657bcc6c2c9460c5d658f48b03497b7 parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679026520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 22:47:01 proc_lsst.multi:170 [INFO] [multi] job_id 17093088 proc_lsst.multi INFO: [multi] job_id 17093088 2024-03-15 22:47:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 22:47:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:01 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093088 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093088 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17093088 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-15 22:47:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-15 22:47:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 22:47:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/ea16de84-f56e-4829-9600-b783556aed93_isrForCrosstalkSources_998106_6.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/ea16de84-f56e-4829-9600-b783556aed93_isrForCrosstalkSources_998106_6.stderr parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash 0a9d1f967b4b810375212a525107f62f parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790263e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/8bd90400-9f77-46c6-a5f8-da10ecc5c8c5_isrForCrosstalkSources_998212_32.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/8bd90400-9f77-46c6-a5f8-da10ecc5c8c5_isrForCrosstalkSources_998212_32.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash a78cdfaede4bb8844f0f361f86c71de7 parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790262a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/99de122a-fde2-4b3e-9fb3-87f66661bd03_isrForCrosstalkSources_998212_4.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/99de122a-fde2-4b3e-9fb3-87f66661bd03_isrForCrosstalkSources_998212_4.stderr parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash c0ced0189010d2888d89fc72154903d9 parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679026480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/cc08f4fa-f04c-4ab7-b1a3-93b2528f612e_isrForCrosstalkSources_998106_7.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/cc08f4fa-f04c-4ab7-b1a3-93b2528f612e_isrForCrosstalkSources_998106_7.stderr parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 5008b29fab2b6ece752fc29729de8f6d parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790249a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/31235152-ece2-4529-be7b-cf32375c45ca_isrForCrosstalkSources_998106_4.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/31235152-ece2-4529-be7b-cf32375c45ca_isrForCrosstalkSources_998106_4.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 5093e491b31ac52bf741c6ba725a85bf parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678eda0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/d18f5946-2ab1-485f-a828-f25cf04e6bf9_isrForCrosstalkSources_998212_17.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/d18f5946-2ab1-485f-a828-f25cf04e6bf9_isrForCrosstalkSources_998212_17.stderr parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash 7eb74515d7337d63ee72e882877c8244 parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679026160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/052e7028-53d4-4ac8-b3e8-7edd19db28e9_isrForCrosstalkSources_998212_52.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/052e7028-53d4-4ac8-b3e8-7edd19db28e9_isrForCrosstalkSources_998212_52.stderr parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash 970061fdb4198afd89bcc1e0b3acce27 parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679026200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/ec2d1151-8cad-45b8-ba52-bd7c8c240135_isrForCrosstalkSources_998211_59.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/ec2d1151-8cad-45b8-ba52-bd7c8c240135_isrForCrosstalkSources_998211_59.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 81125aa8ac0ca1668e03fd0b8bd733e5 parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679025ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/3c629a09-3686-4e42-89c4-1e55520d6512_isrForCrosstalkSources_998210_10.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/3c629a09-3686-4e42-89c4-1e55520d6512_isrForCrosstalkSources_998210_10.stderr parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash 77e18d096d4df9702e197e1d29c9deba parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679025f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/bffe803b-6f78-45d0-b3a8-1360d291a884_isrForCrosstalkSources_998107_24.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/bffe803b-6f78-45d0-b3a8-1360d291a884_isrForCrosstalkSources_998107_24.stderr parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash 47f620b98b22b459f2f35e61af1ed671 parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790260c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/931604c6-088a-4aaa-96ef-eb8ac38009c8_isrForCrosstalkSources_998212_22.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/931604c6-088a-4aaa-96ef-eb8ac38009c8_isrForCrosstalkSources_998212_22.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash c69ba5a9a1f8ea86562591401f7a946f parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679025e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/a9cb2021-aded-4553-a8e7-7b74fbf0648e_isrForCrosstalkSources_998107_48.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/a9cb2021-aded-4553-a8e7-7b74fbf0648e_isrForCrosstalkSources_998107_48.stderr parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash 086f9501efe4df7b20c91b9a0702bed7 parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679025d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/ad8c7e95-9e3d-4e60-ae5d-6671642ec429_isrForCrosstalkSources_998212_47.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/ad8c7e95-9e3d-4e60-ae5d-6671642ec429_isrForCrosstalkSources_998212_47.stderr parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash 6b201125061647c7735bb161aaab1bf8 parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679026020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/90ef2c7b-f237-4f46-8bfb-4dd671ca050c_isrForCrosstalkSources_998210_50.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/90ef2c7b-f237-4f46-8bfb-4dd671ca050c_isrForCrosstalkSources_998210_50.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash cc465582266b157a2132e49834fe336c parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679027100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/cd2a7d75-e55d-4321-9ab6-933742313e25_isrForCrosstalkSources_998107_37.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/cd2a7d75-e55d-4321-9ab6-933742313e25_isrForCrosstalkSources_998107_37.stderr parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash 331616c74750d23eb458553c1acfc05f parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679025da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/3627f4ef-63cf-4ffb-9228-67eef59d2f7d_isrForCrosstalkSources_998211_13.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/3627f4ef-63cf-4ffb-9228-67eef59d2f7d_isrForCrosstalkSources_998211_13.stderr parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash 9eacb3c6ad06efacf41cf1e7352eb9a3 parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679025bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/82358c58-29f5-4600-b362-605f8dc2df43_isrForCrosstalkSources_998107_33.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/82358c58-29f5-4600-b362-605f8dc2df43_isrForCrosstalkSources_998107_33.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash 5946f984c4191e5b1ecaebfb23f997c0 parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679027240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/98a5a088-c14a-4789-afbd-37857a442aff_isrForCrosstalkSources_998106_2.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/98a5a088-c14a-4789-afbd-37857a442aff_isrForCrosstalkSources_998106_2.stderr parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash e137765d9cfd504bb32a2eedffcfcb57 parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790271a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/bade858c-a2e2-4e6c-a471-b4b29936e3e0_isrForCrosstalkSources_998106_40.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/bade858c-a2e2-4e6c-a471-b4b29936e3e0_isrForCrosstalkSources_998106_40.stderr parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash 2605a29bb0cdf8a857b463775b770044 parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679027380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/00c33394-4454-4a6e-ad80-887be6122401_isrForCrosstalkSources_998108_51.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/00c33394-4454-4a6e-ad80-887be6122401_isrForCrosstalkSources_998108_51.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash 1b5650894f6ec96e74368e33ef32d214 parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679027c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/a68d116f-ccb6-495c-b73c-b8095ca66c68_isrForCrosstalkSources_998211_29.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/a68d116f-ccb6-495c-b73c-b8095ca66c68_isrForCrosstalkSources_998211_29.stderr parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash bd1e917ed37a5dc5db1204c92790eb19 parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679027b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/ff96791f-38ed-4c56-b6e5-39a92cd5148d_isrForCrosstalkSources_998210_30.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/ff96791f-38ed-4c56-b6e5-39a92cd5148d_isrForCrosstalkSources_998210_30.stderr parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash ef136381192bf14615c56aacb115bf88 parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679027f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/0427bb26-24e1-4207-8d8a-7eb00a034749_isrForCrosstalkSources_998108_59.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/0427bb26-24e1-4207-8d8a-7eb00a034749_isrForCrosstalkSources_998108_59.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash 7f9071b175986aeddaf727865664796a parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679027a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/8393a5a1-852e-467c-bc49-58e33cba3b83_isrForCrosstalkSources_998210_29.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/8393a5a1-852e-467c-bc49-58e33cba3b83_isrForCrosstalkSources_998210_29.stderr parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash 956d4dd261d35a89046c21b32195e62d parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790279c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/0a305cf0-59c8-49ae-8e65-1bf66e91da17_isrForCrosstalkSources_998106_56.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/0a305cf0-59c8-49ae-8e65-1bf66e91da17_isrForCrosstalkSources_998106_56.stderr parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash 7f1ea70a0b3790c7d2f6b2ac13c7e83e parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679027880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/350f78d5-2c26-4c23-88fe-ef26bddbfe6a_isrForCrosstalkSources_998107_1.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/350f78d5-2c26-4c23-88fe-ef26bddbfe6a_isrForCrosstalkSources_998107_1.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 9dc6882a4fabb853997f125ca6b270f3 parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790277e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/bf94558f-0df6-4680-bc2a-df8e3f6ca360_isrForCrosstalkSources_998211_30.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/bf94558f-0df6-4680-bc2a-df8e3f6ca360_isrForCrosstalkSources_998211_30.stderr parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash 081a34dd7bcb3ed2f17804237f7ba9f6 parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679027920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/226c60a5-7061-4c63-a2b5-2ed1da64fc7a_isrForCrosstalkSources_998106_18.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/226c60a5-7061-4c63-a2b5-2ed1da64fc7a_isrForCrosstalkSources_998106_18.stderr parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash c8d9166c81bb15b19a6bea6e69f02af9 parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679027740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/e5446686-2ceb-4ee9-b64b-1cc6accba819_isrForCrosstalkSources_998107_8.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/e5446686-2ceb-4ee9-b64b-1cc6accba819_isrForCrosstalkSources_998107_8.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash 6490151bfd681b1f9e72c4e550620ba6 parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679027600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/39f8f9a9-df4e-496f-b856-8c7d9712ac15_isrForCrosstalkSources_998210_53.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/39f8f9a9-df4e-496f-b856-8c7d9712ac15_isrForCrosstalkSources_998210_53.stderr parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash 4498724de7df40fcbab6330d77b94e3a parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790276a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/e58e99ff-ff65-4687-ac16-4824b02c8b33_isrForCrosstalkSources_998210_24.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/e58e99ff-ff65-4687-ac16-4824b02c8b33_isrForCrosstalkSources_998210_24.stderr parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash c93e83906c7466eb5cd2c01bf8cf09d5 parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679027560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/25ddb5ba-fa18-4190-9658-a960e0e1b31d_isrForCrosstalkSources_998210_25.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/25ddb5ba-fa18-4190-9658-a960e0e1b31d_isrForCrosstalkSources_998210_25.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash 8dfb18a2b918ce3abce0419efe6c7cae parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790274c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/aafd10cb-7f0e-43ca-b1b6-c8b7c842512f_isrForCrosstalkSources_998108_6.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/aafd10cb-7f0e-43ca-b1b6-c8b7c842512f_isrForCrosstalkSources_998108_6.stderr parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash 096d926f8a3319870edb75e546a30fbc parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679027420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/e05343e7-1be3-4b67-9f67-8052b8c26446_isrForCrosstalkSources_998212_31.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/e05343e7-1be3-4b67-9f67-8052b8c26446_isrForCrosstalkSources_998212_31.stderr parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash a74ec417bfcea40587c9152b0ca3b3f7 parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b6790272e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/45057b27-06b0-464f-8ffb-14302533cd2e_isrForCrosstalkSources_998106_29.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/45057b27-06b0-464f-8ffb-14302533cd2e_isrForCrosstalkSources_998106_29.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash e2e2cc917e72cf08e4ee5774294ac73a parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679026340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 22:47:02 proc_lsst.multi:170 [INFO] [multi] job_id 17093089 proc_lsst.multi INFO: [multi] job_id 17093089 2024-03-15 22:47:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:02 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093089 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093089 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17093089 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-15 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-15 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/9e41ada0-11a8-405b-9879-e8d55bfdb4ba_isrForCrosstalkSources_998210_19.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/9e41ada0-11a8-405b-9879-e8d55bfdb4ba_isrForCrosstalkSources_998210_19.stderr parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash d5ed68f41c73a5096200e2e7ef49cc11 parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679024a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/914fa5ad-44c8-4864-b6fa-c011bc43fc76_isrForCrosstalkSources_998211_35.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/914fa5ad-44c8-4864-b6fa-c011bc43fc76_isrForCrosstalkSources_998211_35.stderr parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash 2d56fb0b10c10d390dd7b4484c2806f3 parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679024ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/124b73cb-6b89-4b75-a822-9218c6095c18_isrForCrosstalkSources_998212_18.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/124b73cb-6b89-4b75-a822-9218c6095c18_isrForCrosstalkSources_998212_18.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash de007bf191289503b4dd845d0249f1a6 parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b679024cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/0a61cd6d-6b47-4d95-8978-def314e2db1b_isrForCrosstalkSources_998210_3.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/0a61cd6d-6b47-4d95-8978-def314e2db1b_isrForCrosstalkSources_998210_3.stderr parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash b30e23b49241fa546fe1382471f4fa5b parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678ed9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/c74debd2-c6ac-45fd-a391-63bf31c86fb6_isrForCrosstalkSources_998211_46.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/c74debd2-c6ac-45fd-a391-63bf31c86fb6_isrForCrosstalkSources_998211_46.stderr parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash 4529a30fe7b4ece4051df4af4c1d3eed parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678ed9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/efaed595-9100-40cd-9f92-b372f0c2c93b_isrForCrosstalkSources_998211_25.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/efaed595-9100-40cd-9f92-b372f0c2c93b_isrForCrosstalkSources_998211_25.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash 68731608689bbf862b24eb5a0eca12c1 parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678ed8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/b151d595-4b74-444f-a567-03ca3718c542_isrForCrosstalkSources_998212_16.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/b151d595-4b74-444f-a567-03ca3718c542_isrForCrosstalkSources_998212_16.stderr parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash 8bc3b576acec5256116f94f5789ec417 parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678ed8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/6856d0dd-c33a-4678-b768-b17f2356440d_isrForCrosstalkSources_998212_44.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/6856d0dd-c33a-4678-b768-b17f2356440d_isrForCrosstalkSources_998212_44.stderr parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash 522102290bd4f680f6f4a1d98629b69c parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678edb880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/da3ec7c1-e4ec-4849-835a-ad93e519f56d_isrForCrosstalkSources_998108_15.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/da3ec7c1-e4ec-4849-835a-ad93e519f56d_isrForCrosstalkSources_998108_15.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash d187295fd1378fc390a722c63c5240a8 parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678edb6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/f4751bb5-8a9b-4b47-a2d2-bd4c204c7d79_isrForCrosstalkSources_998211_54.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/f4751bb5-8a9b-4b47-a2d2-bd4c204c7d79_isrForCrosstalkSources_998211_54.stderr parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash 995a28bb3cbefa09952a9410fdfc904f parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b678ed8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/0ea6c560-6265-421b-b782-0baee2ef971a_isrForCrosstalkSources_998108_19.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/0ea6c560-6265-421b-b782-0baee2ef971a_isrForCrosstalkSources_998108_19.stderr 2024-03-15 22:47:02 proc_lsst.multi:170 [INFO] [multi] job_id 17093090 proc_lsst.multi INFO: [multi] job_id 17093090 2024-03-15 22:47:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-15 22:47:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:02 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093090 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093090 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17093090 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-15 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-15 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-15 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:02 proc_lsst.multi:170 [INFO] [multi] job_id 17093091 proc_lsst.multi INFO: [multi] job_id 17093091 2024-03-15 22:47:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-15 22:47:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:02 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093091 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093091 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17093091 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-15 22:47:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-15 22:47:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-15 22:47:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:03 proc_lsst.multi:170 [INFO] [multi] job_id 17093092 proc_lsst.multi INFO: [multi] job_id 17093092 2024-03-15 22:47:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-15 22:47:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:03 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093092 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093092 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17093092 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-15 22:47:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-15 22:47:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-15 22:47:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:03 proc_lsst.multi:170 [INFO] [multi] job_id 17093093 proc_lsst.multi INFO: [multi] job_id 17093093 2024-03-15 22:47:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-15 22:47:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:03 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093093 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093093 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17093093 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-15 22:47:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-15 22:47:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-15 22:47:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:03 proc_lsst.multi:170 [INFO] [multi] job_id 17093094 proc_lsst.multi INFO: [multi] job_id 17093094 2024-03-15 22:47:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-15 22:47:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:03 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093094 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093094 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17093094 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-15 22:47:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-15 22:47:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-15 22:47:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:04 proc_lsst.multi:170 [INFO] [multi] job_id 17093095 proc_lsst.multi INFO: [multi] job_id 17093095 2024-03-15 22:47:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-15 22:47:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093095 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093095 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17093095 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-15 22:47:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-15 22:47:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-15 22:47:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:04 proc_lsst.multi:170 [INFO] [multi] job_id 17093096 proc_lsst.multi INFO: [multi] job_id 17093096 2024-03-15 22:47:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-15 22:47:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093096 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093096 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17093096 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-15 22:47:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-15 22:47:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-15 22:47:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:04 proc_lsst.multi:170 [INFO] [multi] job_id 17093097 proc_lsst.multi INFO: [multi] job_id 17093097 2024-03-15 22:47:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-15 22:47:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093097 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093097 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17093097 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-15 22:47:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-15 22:47:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-15 22:47:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:05 proc_lsst.multi:170 [INFO] [multi] job_id 17093098 proc_lsst.multi INFO: [multi] job_id 17093098 2024-03-15 22:47:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-15 22:47:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093098 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093098 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17093098 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-15 22:47:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-15 22:47:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-15 22:47:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:05 proc_lsst.multi:170 [INFO] [multi] job_id 17093099 proc_lsst.multi INFO: [multi] job_id 17093099 2024-03-15 22:47:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-15 22:47:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093099 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093099 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17093099 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-15 22:47:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-15 22:47:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-15 22:47:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:06 proc_lsst.multi:170 [INFO] [multi] job_id 17093100 proc_lsst.multi INFO: [multi] job_id 17093100 2024-03-15 22:47:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-15 22:47:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093100 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093100 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17093100 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-15 22:47:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-15 22:47:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-15 22:47:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:06 proc_lsst.multi:170 [INFO] [multi] job_id 17093101 proc_lsst.multi INFO: [multi] job_id 17093101 2024-03-15 22:47:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-15 22:47:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093101 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093101 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17093101 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-15 22:47:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-15 22:47:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-15 22:47:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:06 proc_lsst.multi:170 [INFO] [multi] job_id 17093102 proc_lsst.multi INFO: [multi] job_id 17093102 2024-03-15 22:47:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-15 22:47:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093102 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093102 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17093102 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-15 22:47:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-15 22:47:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-15 22:47:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:07 proc_lsst.multi:170 [INFO] [multi] job_id 17093103 proc_lsst.multi INFO: [multi] job_id 17093103 2024-03-15 22:47:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-15 22:47:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:07 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093103 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093103 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17093103 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-15 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-15 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-15 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:07 proc_lsst.multi:170 [INFO] [multi] job_id 17093104 proc_lsst.multi INFO: [multi] job_id 17093104 2024-03-15 22:47:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:07 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093104 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093104 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17093104 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-15 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-15 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:07 proc_lsst.multi:170 [INFO] [multi] job_id 17093105 proc_lsst.multi INFO: [multi] job_id 17093105 2024-03-15 22:47:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-15 22:47:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:07 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093105 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093105 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17093105 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-15 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-15 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-15 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:07 proc_lsst.multi:170 [INFO] [multi] job_id 17093106 proc_lsst.multi INFO: [multi] job_id 17093106 2024-03-15 22:47:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-15 22:47:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:07 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093106 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093106 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17093106 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-15 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-15 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-15 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:07 proc_lsst.multi:170 [INFO] [multi] job_id 17093107 proc_lsst.multi INFO: [multi] job_id 17093107 2024-03-15 22:47:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-15 22:47:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:07 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093107 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093107 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17093107 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-15 22:47:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-15 22:47:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-15 22:47:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:08 proc_lsst.multi:170 [INFO] [multi] job_id 17093108 proc_lsst.multi INFO: [multi] job_id 17093108 2024-03-15 22:47:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-15 22:47:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093108 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093108 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17093108 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-15 22:47:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-15 22:47:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-15 22:47:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:08 proc_lsst.multi:170 [INFO] [multi] job_id 17093109 proc_lsst.multi INFO: [multi] job_id 17093109 2024-03-15 22:47:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-15 22:47:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093109 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093109 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17093109 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-15 22:47:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-15 22:47:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-15 22:47:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:08 proc_lsst.multi:170 [INFO] [multi] job_id 17093110 proc_lsst.multi INFO: [multi] job_id 17093110 2024-03-15 22:47:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-15 22:47:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093110 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093110 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17093110 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-15 22:47:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-15 22:47:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-15 22:47:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:08 proc_lsst.multi:170 [INFO] [multi] job_id 17093113 proc_lsst.multi INFO: [multi] job_id 17093113 2024-03-15 22:47:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-15 22:47:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093113 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093113 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17093113 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-15 22:47:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-15 22:47:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-15 22:47:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:09 proc_lsst.multi:170 [INFO] [multi] job_id 17093114 proc_lsst.multi INFO: [multi] job_id 17093114 2024-03-15 22:47:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-15 22:47:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093114 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093114 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17093114 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-15 22:47:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-15 22:47:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-15 22:47:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:09 proc_lsst.multi:170 [INFO] [multi] job_id 17093115 proc_lsst.multi INFO: [multi] job_id 17093115 2024-03-15 22:47:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-15 22:47:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093115 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093115 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17093115 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-15 22:47:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-15 22:47:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-15 22:47:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:09 proc_lsst.multi:170 [INFO] [multi] job_id 17093116 proc_lsst.multi INFO: [multi] job_id 17093116 2024-03-15 22:47:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17093116 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17093116 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17093116 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-15 22:47:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-15 22:47:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-15 22:47:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:10 proc_lsst.multi:170 [INFO] [multi] job_id 17093117 proc_lsst.multi INFO: [multi] job_id 17093117 2024-03-15 22:47:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:10 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17093117 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17093117 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17093117 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-15 22:47:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-15 22:47:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:10 proc_lsst.multi:170 [INFO] [multi] job_id 17093118 proc_lsst.multi INFO: [multi] job_id 17093118 2024-03-15 22:47:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 22:47:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:10 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17093118 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17093118 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17093118 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-15 22:47:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-15 22:47:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 22:47:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:10 proc_lsst.multi:170 [INFO] [multi] job_id 17093119 proc_lsst.multi INFO: [multi] job_id 17093119 2024-03-15 22:47:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 22:47:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:10 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17093119 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17093119 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17093119 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-15 22:47:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-15 22:47:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 22:47:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:11 proc_lsst.multi:170 [INFO] [multi] job_id 17093120 proc_lsst.multi INFO: [multi] job_id 17093120 2024-03-15 22:47:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 22:47:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:11 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17093120 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17093120 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17093120 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22773982342992 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 35, active_tasks = 372 parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 1/34 running/pending blocks, and 12 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 35 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 58 more blocks parsl.executors.status_handling INFO: Scaling out by 58 blocks parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-15 22:47:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-15 22:47:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 22:47:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:11 proc_lsst.multi:170 [INFO] [multi] job_id 17093121 proc_lsst.multi INFO: [multi] job_id 17093121 2024-03-15 22:47:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:11 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17093121 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17093121 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17093121 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-15 22:47:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-15 22:47:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-15 22:47:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:11 proc_lsst.multi:170 [INFO] [multi] job_id 17093122 proc_lsst.multi INFO: [multi] job_id 17093122 2024-03-15 22:47:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:11 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17093122 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17093122 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17093122 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-15 22:47:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-15 22:47:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:12 proc_lsst.multi:170 [INFO] [multi] job_id 17093123 proc_lsst.multi INFO: [multi] job_id 17093123 2024-03-15 22:47:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 22:47:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:12 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17093123 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17093123 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17093123 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-15 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-15 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:12 proc_lsst.multi:170 [INFO] [multi] job_id 17093124 proc_lsst.multi INFO: [multi] job_id 17093124 2024-03-15 22:47:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 22:47:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:12 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17093124 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17093124 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17093124 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-15 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-15 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:12 proc_lsst.multi:170 [INFO] [multi] job_id 17093125 proc_lsst.multi INFO: [multi] job_id 17093125 2024-03-15 22:47:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 22:47:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:12 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17093125 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17093125 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17093125 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-15 22:47:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-15 22:47:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 22:47:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9616d5ee877201e2da8e63ac897ea9e with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/4b890ce5-21fa-4544-a798-2e6abb58c5d2_isrForCrosstalkSources_998210_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/4b890ce5-21fa-4544-a798-2e6abb58c5d2_isrForCrosstalkSources_998210_57.stderr 2024-03-15 22:47:13 proc_lsst.multi:170 [INFO] [multi] job_id 17093126 proc_lsst.multi INFO: [multi] job_id 17093126 2024-03-15 22:47:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:13 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17093126 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17093126 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17093126 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-15 22:47:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-15 22:47:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:13 proc_lsst.multi:170 [INFO] [multi] job_id 17093127 proc_lsst.multi INFO: [multi] job_id 17093127 2024-03-15 22:47:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-15 22:47:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:13 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17093127 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17093127 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17093127 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-15 22:47:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-15 22:47:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-15 22:47:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e42459fc873592769083f016a1c7faaa with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/07c43253-3830-4a24-a51c-e01077a3ceda_isrForCrosstalkSources_998212_28.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/07c43253-3830-4a24-a51c-e01077a3ceda_isrForCrosstalkSources_998212_28.stderr 2024-03-15 22:47:13 proc_lsst.multi:170 [INFO] [multi] job_id 17093128 proc_lsst.multi INFO: [multi] job_id 17093128 2024-03-15 22:47:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-15 22:47:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:13 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17093128 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17093128 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17093128 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-15 22:47:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-15 22:47:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-15 22:47:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:13 proc_lsst.multi:170 [INFO] [multi] job_id 17093129 proc_lsst.multi INFO: [multi] job_id 17093129 2024-03-15 22:47:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-15 22:47:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:13 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17093129 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17093129 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17093129 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-15 22:47:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-15 22:47:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-15 22:47:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:14 proc_lsst.multi:170 [INFO] [multi] job_id 17093130 proc_lsst.multi INFO: [multi] job_id 17093130 2024-03-15 22:47:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-15 22:47:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:14 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17093130 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17093130 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17093130 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-15 22:47:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-15 22:47:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-15 22:47:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:14 proc_lsst.multi:170 [INFO] [multi] job_id 17093132 proc_lsst.multi INFO: [multi] job_id 17093132 2024-03-15 22:47:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-15 22:47:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:14 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17093132 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17093132 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17093132 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-15 22:47:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-15 22:47:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-15 22:47:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:14 proc_lsst.multi:170 [INFO] [multi] job_id 17093133 proc_lsst.multi INFO: [multi] job_id 17093133 2024-03-15 22:47:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-15 22:47:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:14 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17093133 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17093133 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 17093133 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-15 22:47:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-15 22:47:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-15 22:47:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:15 proc_lsst.multi:170 [INFO] [multi] job_id 17093135 proc_lsst.multi INFO: [multi] job_id 17093135 2024-03-15 22:47:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-15 22:47:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:15 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17093135 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17093135 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 17093135 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-15 22:47:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-15 22:47:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-15 22:47:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:15 proc_lsst.multi:170 [INFO] [multi] job_id 17093136 proc_lsst.multi INFO: [multi] job_id 17093136 2024-03-15 22:47:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-15 22:47:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:15 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17093136 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17093136 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 17093136 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-15 22:47:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-15 22:47:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-15 22:47:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:15 proc_lsst.multi:170 [INFO] [multi] job_id 17093138 proc_lsst.multi INFO: [multi] job_id 17093138 2024-03-15 22:47:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-15 22:47:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:15 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17093138 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17093138 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 17093138 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-15 22:47:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-15 22:47:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-15 22:47:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:16 proc_lsst.multi:170 [INFO] [multi] job_id 17093140 proc_lsst.multi INFO: [multi] job_id 17093140 2024-03-15 22:47:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-15 22:47:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:16 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17093140 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17093140 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 17093140 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-15 22:47:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-15 22:47:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-15 22:47:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:16 proc_lsst.multi:170 [INFO] [multi] job_id 17093142 proc_lsst.multi INFO: [multi] job_id 17093142 2024-03-15 22:47:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-15 22:47:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:16 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17093142 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17093142 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 17093142 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-15 22:47:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-15 22:47:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-15 22:47:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:18 proc_lsst.multi:170 [INFO] [multi] job_id 17093143 proc_lsst.multi INFO: [multi] job_id 17093143 2024-03-15 22:47:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-15 22:47:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:18 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17093143 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17093143 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 17093143 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-15 22:47:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-15 22:47:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-15 22:47:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:18 proc_lsst.multi:170 [INFO] [multi] job_id 17093145 proc_lsst.multi INFO: [multi] job_id 17093145 2024-03-15 22:47:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-15 22:47:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:18 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17093145 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17093145 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 17093145 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-15 22:47:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-15 22:47:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-15 22:47:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:18 proc_lsst.multi:170 [INFO] [multi] job_id 17093146 proc_lsst.multi INFO: [multi] job_id 17093146 2024-03-15 22:47:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-15 22:47:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:18 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17093146 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17093146 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 17093146 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-15 22:47:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-15 22:47:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-15 22:47:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:19 proc_lsst.multi:170 [INFO] [multi] job_id 17093147 proc_lsst.multi INFO: [multi] job_id 17093147 2024-03-15 22:47:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:19 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17093147 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17093147 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 17093147 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-15 22:47:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:19 proc_lsst.multi:170 [INFO] [multi] job_id 17093149 proc_lsst.multi INFO: [multi] job_id 17093149 2024-03-15 22:47:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093149 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093149 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 17093149 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-15 22:47:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:19 proc_lsst.multi:170 [INFO] [multi] job_id 17093150 proc_lsst.multi INFO: [multi] job_id 17093150 2024-03-15 22:47:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 22:47:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093150 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093150 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 17093150 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-15 22:47:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:19 proc_lsst.multi:170 [INFO] [multi] job_id 17093151 proc_lsst.multi INFO: [multi] job_id 17093151 2024-03-15 22:47:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 22:47:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093151 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093151 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 17093151 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-15 22:47:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:19 proc_lsst.multi:170 [INFO] [multi] job_id 17093153 proc_lsst.multi INFO: [multi] job_id 17093153 2024-03-15 22:47:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 22:47:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093153 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093153 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 17093153 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-15 22:47:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:19 proc_lsst.multi:170 [INFO] [multi] job_id 17093154 proc_lsst.multi INFO: [multi] job_id 17093154 2024-03-15 22:47:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093154 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093154 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 17093154 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-15 22:47:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:20 proc_lsst.multi:170 [INFO] [multi] job_id 17093155 proc_lsst.multi INFO: [multi] job_id 17093155 2024-03-15 22:47:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-15 22:47:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093155 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093155 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 17093155 parsl.executors.status_handling INFO: Allocated block ID 62 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62 2024-03-15 22:47:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 2024-03-15 22:47:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-15 22:47:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:20 proc_lsst.multi:170 [INFO] [multi] job_id 17093157 proc_lsst.multi INFO: [multi] job_id 17093157 2024-03-15 22:47:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-15 22:47:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093157 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093157 parsl.executors.status_handling DEBUG: Launched block 62 on executor multi with job ID 17093157 parsl.executors.status_handling INFO: Allocated block ID 63 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63 2024-03-15 22:47:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 2024-03-15 22:47:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-15 22:47:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:20 proc_lsst.multi:170 [INFO] [multi] job_id 17093158 proc_lsst.multi INFO: [multi] job_id 17093158 2024-03-15 22:47:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-15 22:47:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093158 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093158 parsl.executors.status_handling DEBUG: Launched block 63 on executor multi with job ID 17093158 parsl.executors.status_handling INFO: Allocated block ID 64 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64 2024-03-15 22:47:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 2024-03-15 22:47:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-15 22:47:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:20 proc_lsst.multi:170 [INFO] [multi] job_id 17093159 proc_lsst.multi INFO: [multi] job_id 17093159 2024-03-15 22:47:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-15 22:47:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093159 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093159 parsl.executors.status_handling DEBUG: Launched block 64 on executor multi with job ID 17093159 parsl.executors.status_handling INFO: Allocated block ID 65 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65 2024-03-15 22:47:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 2024-03-15 22:47:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-15 22:47:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:21 proc_lsst.multi:170 [INFO] [multi] job_id 17093160 proc_lsst.multi INFO: [multi] job_id 17093160 2024-03-15 22:47:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-15 22:47:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093160 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093160 parsl.executors.status_handling DEBUG: Launched block 65 on executor multi with job ID 17093160 parsl.executors.status_handling INFO: Allocated block ID 66 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66 2024-03-15 22:47:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 2024-03-15 22:47:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-15 22:47:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:21 proc_lsst.multi:170 [INFO] [multi] job_id 17093161 proc_lsst.multi INFO: [multi] job_id 17093161 2024-03-15 22:47:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-15 22:47:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093161 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093161 parsl.executors.status_handling DEBUG: Launched block 66 on executor multi with job ID 17093161 parsl.executors.status_handling INFO: Allocated block ID 67 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67 2024-03-15 22:47:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 2024-03-15 22:47:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-15 22:47:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:21 proc_lsst.multi:170 [INFO] [multi] job_id 17093162 proc_lsst.multi INFO: [multi] job_id 17093162 2024-03-15 22:47:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-15 22:47:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093162 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093162 parsl.executors.status_handling DEBUG: Launched block 67 on executor multi with job ID 17093162 parsl.executors.status_handling INFO: Allocated block ID 68 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68 2024-03-15 22:47:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 2024-03-15 22:47:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-15 22:47:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:22 proc_lsst.multi:170 [INFO] [multi] job_id 17093163 proc_lsst.multi INFO: [multi] job_id 17093163 2024-03-15 22:47:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-15 22:47:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093163 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093163 parsl.executors.status_handling DEBUG: Launched block 68 on executor multi with job ID 17093163 parsl.executors.status_handling INFO: Allocated block ID 69 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69 2024-03-15 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 2024-03-15 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-15 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:22 proc_lsst.multi:170 [INFO] [multi] job_id 17093164 proc_lsst.multi INFO: [multi] job_id 17093164 2024-03-15 22:47:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-15 22:47:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093164 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093164 parsl.executors.status_handling DEBUG: Launched block 69 on executor multi with job ID 17093164 parsl.executors.status_handling INFO: Allocated block ID 70 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70 2024-03-15 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 2024-03-15 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-15 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:22 proc_lsst.multi:170 [INFO] [multi] job_id 17093165 proc_lsst.multi INFO: [multi] job_id 17093165 2024-03-15 22:47:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-15 22:47:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093165 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093165 parsl.executors.status_handling DEBUG: Launched block 70 on executor multi with job ID 17093165 parsl.executors.status_handling INFO: Allocated block ID 71 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71 2024-03-15 22:47:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 2024-03-15 22:47:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-15 22:47:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:23 proc_lsst.multi:170 [INFO] [multi] job_id 17093166 proc_lsst.multi INFO: [multi] job_id 17093166 2024-03-15 22:47:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-15 22:47:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093166 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093166 parsl.executors.status_handling DEBUG: Launched block 71 on executor multi with job ID 17093166 parsl.executors.status_handling INFO: Allocated block ID 72 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72 2024-03-15 22:47:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 2024-03-15 22:47:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-15 22:47:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a28ae174b17b37086690ff0841300c3b with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/7bb75482-bccc-45a4-acbb-e0c383d2e336_isrForCrosstalkSources_998108_12.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/7bb75482-bccc-45a4-acbb-e0c383d2e336_isrForCrosstalkSources_998108_12.stderr 2024-03-15 22:47:23 proc_lsst.multi:170 [INFO] [multi] job_id 17093167 proc_lsst.multi INFO: [multi] job_id 17093167 2024-03-15 22:47:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-15 22:47:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093167 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093167 parsl.executors.status_handling DEBUG: Launched block 72 on executor multi with job ID 17093167 parsl.executors.status_handling INFO: Allocated block ID 73 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73 2024-03-15 22:47:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 2024-03-15 22:47:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-15 22:47:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:23 proc_lsst.multi:170 [INFO] [multi] job_id 17093168 proc_lsst.multi INFO: [multi] job_id 17093168 2024-03-15 22:47:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-15 22:47:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093168 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093168 parsl.executors.status_handling DEBUG: Launched block 73 on executor multi with job ID 17093168 parsl.executors.status_handling INFO: Allocated block ID 74 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74 2024-03-15 22:47:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 2024-03-15 22:47:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-15 22:47:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:24 proc_lsst.multi:170 [INFO] [multi] job_id 17093169 proc_lsst.multi INFO: [multi] job_id 17093169 2024-03-15 22:47:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-15 22:47:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093169 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093169 parsl.executors.status_handling DEBUG: Launched block 74 on executor multi with job ID 17093169 parsl.executors.status_handling INFO: Allocated block ID 75 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75 2024-03-15 22:47:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 2024-03-15 22:47:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-15 22:47:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:24 proc_lsst.multi:170 [INFO] [multi] job_id 17093170 proc_lsst.multi INFO: [multi] job_id 17093170 2024-03-15 22:47:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093170 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093170 parsl.executors.status_handling DEBUG: Launched block 75 on executor multi with job ID 17093170 parsl.executors.status_handling INFO: Allocated block ID 76 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76 2024-03-15 22:47:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 2024-03-15 22:47:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:24 proc_lsst.multi:170 [INFO] [multi] job_id 17093171 proc_lsst.multi INFO: [multi] job_id 17093171 2024-03-15 22:47:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-15 22:47:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093171 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093171 parsl.executors.status_handling DEBUG: Launched block 76 on executor multi with job ID 17093171 parsl.executors.status_handling INFO: Allocated block ID 77 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77 2024-03-15 22:47:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 2024-03-15 22:47:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-15 22:47:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:24 proc_lsst.multi:170 [INFO] [multi] job_id 17093172 proc_lsst.multi INFO: [multi] job_id 17093172 2024-03-15 22:47:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-15 22:47:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093172 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093172 parsl.executors.status_handling DEBUG: Launched block 77 on executor multi with job ID 17093172 parsl.executors.status_handling INFO: Allocated block ID 78 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-78 2024-03-15 22:47:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 2024-03-15 22:47:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-15 22:47:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:25 proc_lsst.multi:170 [INFO] [multi] job_id 17093173 proc_lsst.multi INFO: [multi] job_id 17093173 2024-03-15 22:47:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-15 22:47:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093173 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093173 parsl.executors.status_handling DEBUG: Launched block 78 on executor multi with job ID 17093173 parsl.executors.status_handling INFO: Allocated block ID 79 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-79 2024-03-15 22:47:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 2024-03-15 22:47:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-15 22:47:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c4fa9feca951231fa2f03f9e655042b with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/104648bd-b818-4c56-877d-8d738fc195c7_isrForCrosstalkSources_998107_58.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/104648bd-b818-4c56-877d-8d738fc195c7_isrForCrosstalkSources_998107_58.stderr 2024-03-15 22:47:25 proc_lsst.multi:170 [INFO] [multi] job_id 17093174 proc_lsst.multi INFO: [multi] job_id 17093174 2024-03-15 22:47:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-15 22:47:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093174 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093174 parsl.executors.status_handling DEBUG: Launched block 79 on executor multi with job ID 17093174 parsl.executors.status_handling INFO: Allocated block ID 80 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-80 2024-03-15 22:47:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 2024-03-15 22:47:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-15 22:47:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:25 proc_lsst.multi:170 [INFO] [multi] job_id 17093175 proc_lsst.multi INFO: [multi] job_id 17093175 2024-03-15 22:47:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-15 22:47:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093175 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093175 parsl.executors.status_handling DEBUG: Launched block 80 on executor multi with job ID 17093175 parsl.executors.status_handling INFO: Allocated block ID 81 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-81 2024-03-15 22:47:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 2024-03-15 22:47:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-15 22:47:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:25 proc_lsst.multi:170 [INFO] [multi] job_id 17093176 proc_lsst.multi INFO: [multi] job_id 17093176 2024-03-15 22:47:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-15 22:47:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093176 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093176 parsl.executors.status_handling DEBUG: Launched block 81 on executor multi with job ID 17093176 parsl.executors.status_handling INFO: Allocated block ID 82 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-82 2024-03-15 22:47:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 2024-03-15 22:47:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-15 22:47:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:26 proc_lsst.multi:170 [INFO] [multi] job_id 17093177 proc_lsst.multi INFO: [multi] job_id 17093177 2024-03-15 22:47:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-15 22:47:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093177 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093177 parsl.executors.status_handling DEBUG: Launched block 82 on executor multi with job ID 17093177 parsl.executors.status_handling INFO: Allocated block ID 83 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-83 2024-03-15 22:47:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 2024-03-15 22:47:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-15 22:47:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:26 proc_lsst.multi:170 [INFO] [multi] job_id 17093178 proc_lsst.multi INFO: [multi] job_id 17093178 2024-03-15 22:47:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-15 22:47:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093178 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093178 parsl.executors.status_handling DEBUG: Launched block 83 on executor multi with job ID 17093178 parsl.executors.status_handling INFO: Allocated block ID 84 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-84 2024-03-15 22:47:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 2024-03-15 22:47:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-15 22:47:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:26 proc_lsst.multi:170 [INFO] [multi] job_id 17093179 proc_lsst.multi INFO: [multi] job_id 17093179 2024-03-15 22:47:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-15 22:47:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093179 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093179 parsl.executors.status_handling DEBUG: Launched block 84 on executor multi with job ID 17093179 parsl.executors.status_handling INFO: Allocated block ID 85 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-85 2024-03-15 22:47:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 2024-03-15 22:47:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-15 22:47:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:26 proc_lsst.multi:170 [INFO] [multi] job_id 17093180 proc_lsst.multi INFO: [multi] job_id 17093180 2024-03-15 22:47:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093180 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093180 parsl.executors.status_handling DEBUG: Launched block 85 on executor multi with job ID 17093180 parsl.executors.status_handling INFO: Allocated block ID 86 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-86 2024-03-15 22:47:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 2024-03-15 22:47:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:27 proc_lsst.multi:170 [INFO] [multi] job_id 17093181 proc_lsst.multi INFO: [multi] job_id 17093181 2024-03-15 22:47:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-15 22:47:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093181 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093181 parsl.executors.status_handling DEBUG: Launched block 86 on executor multi with job ID 17093181 parsl.executors.status_handling INFO: Allocated block ID 87 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-87 2024-03-15 22:47:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 2024-03-15 22:47:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-15 22:47:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:27 proc_lsst.multi:170 [INFO] [multi] job_id 17093182 proc_lsst.multi INFO: [multi] job_id 17093182 2024-03-15 22:47:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-15 22:47:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093182 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093182 parsl.executors.status_handling DEBUG: Launched block 87 on executor multi with job ID 17093182 parsl.executors.status_handling INFO: Allocated block ID 88 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-88 2024-03-15 22:47:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 2024-03-15 22:47:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-15 22:47:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:28 proc_lsst.multi:170 [INFO] [multi] job_id 17093183 proc_lsst.multi INFO: [multi] job_id 17093183 2024-03-15 22:47:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-15 22:47:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093183 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093183 parsl.executors.status_handling DEBUG: Launched block 88 on executor multi with job ID 17093183 parsl.executors.status_handling INFO: Allocated block ID 89 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-89 2024-03-15 22:47:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 2024-03-15 22:47:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-15 22:47:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:28 proc_lsst.multi:170 [INFO] [multi] job_id 17093184 proc_lsst.multi INFO: [multi] job_id 17093184 2024-03-15 22:47:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-15 22:47:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093184 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093184 parsl.executors.status_handling DEBUG: Launched block 89 on executor multi with job ID 17093184 parsl.executors.status_handling INFO: Allocated block ID 90 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-90 2024-03-15 22:47:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 2024-03-15 22:47:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-15 22:47:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:29 proc_lsst.multi:170 [INFO] [multi] job_id 17093185 proc_lsst.multi INFO: [multi] job_id 17093185 2024-03-15 22:47:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-15 22:47:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093185 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093185 parsl.executors.status_handling DEBUG: Launched block 90 on executor multi with job ID 17093185 parsl.executors.status_handling INFO: Allocated block ID 91 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-91 2024-03-15 22:47:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 2024-03-15 22:47:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-15 22:47:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:29 proc_lsst.multi:170 [INFO] [multi] job_id 17093186 proc_lsst.multi INFO: [multi] job_id 17093186 2024-03-15 22:47:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-15 22:47:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093186 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093186 parsl.executors.status_handling DEBUG: Launched block 91 on executor multi with job ID 17093186 parsl.executors.status_handling INFO: Allocated block ID 92 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-92 2024-03-15 22:47:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 2024-03-15 22:47:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 22:47:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 22:47:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 22:47:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 22:47:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 22:47:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 22:47:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 22:47:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 22:47:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 22:47:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 22:47:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-15 22:47:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54311 --result_port=54449 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 22:47:30 proc_lsst.multi:170 [INFO] [multi] job_id 17093187 proc_lsst.multi INFO: [multi] job_id 17093187 2024-03-15 22:47:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 37 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37 2024-03-15 22:47:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 22:47:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17093187 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17093187 parsl.executors.status_handling DEBUG: Launched block 92 on executor multi with job ID 17093187 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22773982342992 2024-03-15 22:47:30 proc_lsst.multi:146 [INFO] found job 36385 in provider local proc_lsst.multi INFO: found job 36385 in provider local 2024-03-15 22:47:30 proc_lsst.multi:146 [INFO] found job 17093085 in provider astro proc_lsst.multi INFO: found job 17093085 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:30 proc_lsst.multi:146 [INFO] found job 17093086 in provider astro proc_lsst.multi INFO: found job 17093086 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:31 proc_lsst.multi:146 [INFO] found job 17093087 in provider astro proc_lsst.multi INFO: found job 17093087 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:31 proc_lsst.multi:146 [INFO] found job 17093088 in provider astro proc_lsst.multi INFO: found job 17093088 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:31 proc_lsst.multi:146 [INFO] found job 17093089 in provider astro proc_lsst.multi INFO: found job 17093089 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:31 proc_lsst.multi:146 [INFO] found job 17093090 in provider astro proc_lsst.multi INFO: found job 17093090 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:31 proc_lsst.multi:146 [INFO] found job 17093091 in provider astro proc_lsst.multi INFO: found job 17093091 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:32 proc_lsst.multi:146 [INFO] found job 17093092 in provider astro proc_lsst.multi INFO: found job 17093092 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:32 proc_lsst.multi:146 [INFO] found job 17093093 in provider astro proc_lsst.multi INFO: found job 17093093 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:33 proc_lsst.multi:146 [INFO] found job 17093094 in provider astro proc_lsst.multi INFO: found job 17093094 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:33 proc_lsst.multi:146 [INFO] found job 17093095 in provider astro proc_lsst.multi INFO: found job 17093095 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:34 proc_lsst.multi:146 [INFO] found job 17093096 in provider astro proc_lsst.multi INFO: found job 17093096 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:34 proc_lsst.multi:146 [INFO] found job 17093097 in provider astro proc_lsst.multi INFO: found job 17093097 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:35 proc_lsst.multi:146 [INFO] found job 17093098 in provider astro proc_lsst.multi INFO: found job 17093098 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:35 proc_lsst.multi:146 [INFO] found job 17093099 in provider astro proc_lsst.multi INFO: found job 17093099 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6f15f8f34c41e35b5e70d0adc12c13b with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/c2a47194-c054-4a8c-aa53-96868430049d_isrForCrosstalkSources_998106_5.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/c2a47194-c054-4a8c-aa53-96868430049d_isrForCrosstalkSources_998106_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:36 proc_lsst.multi:146 [INFO] found job 17093100 in provider astro proc_lsst.multi INFO: found job 17093100 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:36 proc_lsst.multi:146 [INFO] found job 17093101 in provider astro proc_lsst.multi INFO: found job 17093101 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:37 proc_lsst.multi:146 [INFO] found job 17093102 in provider astro proc_lsst.multi INFO: found job 17093102 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56705a745b50fd0ecd74022f9e8c94e1 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/d436dac6-bbc1-491d-a5d7-d112d0450032_isrForCrosstalkSources_998212_1.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/d436dac6-bbc1-491d-a5d7-d112d0450032_isrForCrosstalkSources_998212_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:37 proc_lsst.multi:146 [INFO] found job 17093103 in provider astro proc_lsst.multi INFO: found job 17093103 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:37 proc_lsst.multi:146 [INFO] found job 17093104 in provider astro proc_lsst.multi INFO: found job 17093104 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:38 proc_lsst.multi:146 [INFO] found job 17093105 in provider astro proc_lsst.multi INFO: found job 17093105 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:38 proc_lsst.multi:146 [INFO] found job 17093106 in provider astro proc_lsst.multi INFO: found job 17093106 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:39 proc_lsst.multi:146 [INFO] found job 17093107 in provider astro proc_lsst.multi INFO: found job 17093107 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:39 proc_lsst.multi:146 [INFO] found job 17093108 in provider astro proc_lsst.multi INFO: found job 17093108 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:40 proc_lsst.multi:146 [INFO] found job 17093109 in provider astro proc_lsst.multi INFO: found job 17093109 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:40 proc_lsst.multi:146 [INFO] found job 17093110 in provider astro proc_lsst.multi INFO: found job 17093110 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:41 proc_lsst.multi:146 [INFO] found job 17093113 in provider astro proc_lsst.multi INFO: found job 17093113 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:41 proc_lsst.multi:146 [INFO] found job 17093114 in provider astro proc_lsst.multi INFO: found job 17093114 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:41 proc_lsst.multi:146 [INFO] found job 17093115 in provider astro proc_lsst.multi INFO: found job 17093115 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:42 proc_lsst.multi:146 [INFO] found job 17093116 in provider astro proc_lsst.multi INFO: found job 17093116 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:42 proc_lsst.multi:146 [INFO] found job 17093117 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093117 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093117,17093118,17093119,17093120,17093121' parsl.providers.slurm.slurm DEBUG: squeue returned 17093117 R 17093118 R 17093119 R 17093120 R 17093121 R parsl.providers.slurm.slurm DEBUG: Updating job 17093117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093121 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:43 proc_lsst.multi:146 [INFO] found job 17093118 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093118 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093117,17093118,17093119,17093120,17093121' parsl.providers.slurm.slurm DEBUG: squeue returned 17093117 R 17093118 R 17093119 R 17093120 R 17093121 R parsl.providers.slurm.slurm DEBUG: Updating job 17093117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093121 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:43 proc_lsst.multi:146 [INFO] found job 17093119 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093119 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093117,17093118,17093119,17093120,17093121' parsl.providers.slurm.slurm DEBUG: squeue returned 17093117 R 17093118 R 17093119 R 17093120 R 17093121 R parsl.providers.slurm.slurm DEBUG: Updating job 17093117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093121 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:44 proc_lsst.multi:146 [INFO] found job 17093120 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093120 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093117,17093118,17093119,17093120,17093121' parsl.providers.slurm.slurm DEBUG: squeue returned 17093117 R 17093118 R 17093119 R 17093120 R 17093121 R parsl.providers.slurm.slurm DEBUG: Updating job 17093117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093121 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:44 proc_lsst.multi:146 [INFO] found job 17093121 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093121 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093117,17093118,17093119,17093120,17093121' parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fdf4f7a1a3ac9fd1c2ae717a9680e90 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/3c99bca7-cb56-4b2c-8c9b-5fd2d246ef1f_isrForCrosstalkSources_998210_6.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/3c99bca7-cb56-4b2c-8c9b-5fd2d246ef1f_isrForCrosstalkSources_998210_6.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4177a54f59de07d813c035eb5d9993d3 with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/57bb81d5-5d22-4142-b614-12efca3cb663_isrForCrosstalkSources_998212_40.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/57bb81d5-5d22-4142-b614-12efca3cb663_isrForCrosstalkSources_998212_40.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54c9ef240cc917899e78b6963af6d0bd with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/7f33b8b1-f5f9-47c0-a407-7b2a7be3126c_isrForCrosstalkSources_998212_23.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/7f33b8b1-f5f9-47c0-a407-7b2a7be3126c_isrForCrosstalkSources_998212_23.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 575fc68a6bc63dd60428f38c0cebc726 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/6614a1a1-a992-46b4-923e-ef703911d2d7_isrForCrosstalkSources_998212_57.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/6614a1a1-a992-46b4-923e-ef703911d2d7_isrForCrosstalkSources_998212_57.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 926bcfe3ae80a9052987669e45543be3 with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/241954d5-1a3c-444f-971f-5be30acb5e08_isrForCrosstalkSources_998108_16.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/241954d5-1a3c-444f-971f-5be30acb5e08_isrForCrosstalkSources_998108_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093117 R 17093118 R 17093119 R 17093120 R 17093121 R parsl.providers.slurm.slurm DEBUG: Updating job 17093117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093121 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:45 proc_lsst.multi:146 [INFO] found job 17093122 in provider escience_a40 proc_lsst.multi INFO: found job 17093122 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:45 proc_lsst.multi:146 [INFO] found job 17093123 in provider escience_a40 proc_lsst.multi INFO: found job 17093123 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:45 proc_lsst.multi:146 [INFO] found job 17093124 in provider escience_a40 proc_lsst.multi INFO: found job 17093124 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac698e69abc75ab03a2eb79ff0379590 with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/3df08b00-0815-48f9-942a-a732b26f2985_isrForCrosstalkSources_998107_56.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/3df08b00-0815-48f9-942a-a732b26f2985_isrForCrosstalkSources_998107_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:46 proc_lsst.multi:146 [INFO] found job 17093125 in provider escience_a40 proc_lsst.multi INFO: found job 17093125 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9e33010f7db2f311b77176941f1b8de with result from task 62 parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:46 proc_lsst.multi:146 [INFO] found job 17093126 in provider escience_a40 proc_lsst.multi INFO: found job 17093126 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/3cc721a9-33eb-4577-9e03-d70522f4f59b_isrForCrosstalkSources_998106_51.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/3cc721a9-33eb-4577-9e03-d70522f4f59b_isrForCrosstalkSources_998106_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:47 proc_lsst.multi:146 [INFO] found job 17093127 in provider escience_a40 proc_lsst.multi INFO: found job 17093127 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:47 proc_lsst.multi:146 [INFO] found job 17093128 in provider escience_a40 proc_lsst.multi INFO: found job 17093128 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:48 proc_lsst.multi:146 [INFO] found job 17093129 in provider escience_a40 proc_lsst.multi INFO: found job 17093129 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:48 proc_lsst.multi:146 [INFO] found job 17093130 in provider escience_a40 proc_lsst.multi INFO: found job 17093130 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff5cde6d01021c229eceb14d30e21222 with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/fe72e78a-1580-41d7-a079-c3016a31193d_isrForCrosstalkSources_998212_60.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/fe72e78a-1580-41d7-a079-c3016a31193d_isrForCrosstalkSources_998212_60.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4abc58e7db00a5ca967f1f34d9d613d7 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/0a43a729-7352-4685-a26a-3d533aa3fc16_isrForCrosstalkSources_998212_8.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/0a43a729-7352-4685-a26a-3d533aa3fc16_isrForCrosstalkSources_998212_8.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54bf201f70cc452b1a2270dd7e2f9f1a with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/208fb8dd-2f78-4109-bd3c-4345b9a56fdf_isrForCrosstalkSources_998212_21.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/208fb8dd-2f78-4109-bd3c-4345b9a56fdf_isrForCrosstalkSources_998212_21.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9559faff1c9f8d4cf47314eff9628fa4 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/c584b445-996a-4b11-a3e8-c66a971bdf5a_isrForCrosstalkSources_998106_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/c584b445-996a-4b11-a3e8-c66a971bdf5a_isrForCrosstalkSources_998106_55.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c96e0ace76eb6d92dcad2a0b1e143dc8 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/57fc39b6-abb5-48f3-b49c-e0cc60da6897_isrForCrosstalkSources_998108_38.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/57fc39b6-abb5-48f3-b49c-e0cc60da6897_isrForCrosstalkSources_998108_38.stderr parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89b42038f090c0f79e1f52a364ea4de8 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/42ef80d4-e9aa-4eb8-9a14-49baa9be223a_isrForCrosstalkSources_998106_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/42ef80d4-e9aa-4eb8-9a14-49baa9be223a_isrForCrosstalkSources_998106_30.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62c374128f647751be08e92df88b18a6 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/491f0ccf-51a9-476c-bd8c-1dc89f5114b1_isrForCrosstalkSources_998106_46.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/491f0ccf-51a9-476c-bd8c-1dc89f5114b1_isrForCrosstalkSources_998106_46.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 332f383fed125d439b89bdd88c5a5bbd with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/c56b6e7a-2a6b-47fd-b407-5eed9200b065_isrForCrosstalkSources_998106_54.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/c56b6e7a-2a6b-47fd-b407-5eed9200b065_isrForCrosstalkSources_998106_54.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad53e568cab260599ab232719993de45 with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/66e59bf4-6dee-4246-92ec-963029ae362f_isrForCrosstalkSources_998211_37.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/66e59bf4-6dee-4246-92ec-963029ae362f_isrForCrosstalkSources_998211_37.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c825007bdc92f0162dc45ec81556e0e9 with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/d782e0c0-a046-44ce-a435-73d2aedaa1c3_isrForCrosstalkSources_998211_6.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/d782e0c0-a046-44ce-a435-73d2aedaa1c3_isrForCrosstalkSources_998211_6.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7002ef7f26b1856c33e79dc43a4d5845 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/cbe910f3-3c43-4025-beae-e986943ef6ec_isrForCrosstalkSources_998108_26.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/cbe910f3-3c43-4025-beae-e986943ef6ec_isrForCrosstalkSources_998108_26.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6220f105075b611c2b868190c7532cb with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/6f041b6f-b1fc-4ed5-8a57-115d3398399b_isrForCrosstalkSources_998107_20.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/6f041b6f-b1fc-4ed5-8a57-115d3398399b_isrForCrosstalkSources_998107_20.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08bcabe048746f733c623e17ee566920 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/1a870a33-df08-4718-82fb-8ce6e7faef9b_isrForCrosstalkSources_998106_27.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/1a870a33-df08-4718-82fb-8ce6e7faef9b_isrForCrosstalkSources_998106_27.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37d9b2051880121a761dffc13a7fe53b with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/076f97fb-11ce-4a56-b71e-aece0a6fcda8_isrForCrosstalkSources_998211_62.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/076f97fb-11ce-4a56-b71e-aece0a6fcda8_isrForCrosstalkSources_998211_62.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c71d5e25d11c54b9ad899516f2b2ba7 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/b9776b41-3387-4142-ac99-68d81ecafde2_isrForCrosstalkSources_998212_61.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/b9776b41-3387-4142-ac99-68d81ecafde2_isrForCrosstalkSources_998212_61.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d638136d663b70c4655c2b36ad4369ba with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/aa615a90-8aa8-42ce-a8d9-ebbbd8b399be_isrForCrosstalkSources_998108_37.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/aa615a90-8aa8-42ce-a8d9-ebbbd8b399be_isrForCrosstalkSources_998108_37.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4fdaf1d23b941edba8627fca38e2ffa4 with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/aaa9a1d9-8619-4d67-ae24-4000247904f7_isrForCrosstalkSources_998210_18.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/aaa9a1d9-8619-4d67-ae24-4000247904f7_isrForCrosstalkSources_998210_18.stderr parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fde8a4728c18950b75c2adabbd405b7d with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/e8da4bb8-ae6d-43e3-8bc9-840e356a48be_isrForCrosstalkSources_998211_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/e8da4bb8-ae6d-43e3-8bc9-840e356a48be_isrForCrosstalkSources_998211_57.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb1a5dc50e5531e1f26c0c81bd4323b3 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/1463b5cb-768c-401c-87bb-5a84fefb2744_isrForCrosstalkSources_998106_25.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/1463b5cb-768c-401c-87bb-5a84fefb2744_isrForCrosstalkSources_998106_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:50 proc_lsst.multi:146 [INFO] found job 17093132 in provider escience_a40 proc_lsst.multi INFO: found job 17093132 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:51 proc_lsst.multi:146 [INFO] found job 17093133 in provider escience_a40 proc_lsst.multi INFO: found job 17093133 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:52 proc_lsst.multi:146 [INFO] found job 17093135 in provider escience_a40 proc_lsst.multi INFO: found job 17093135 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:52 proc_lsst.multi:146 [INFO] found job 17093136 in provider escience_a40 proc_lsst.multi INFO: found job 17093136 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:53 proc_lsst.multi:146 [INFO] found job 17093138 in provider escience_a40 proc_lsst.multi INFO: found job 17093138 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:54 proc_lsst.multi:146 [INFO] found job 17093140 in provider escience_a40 proc_lsst.multi INFO: found job 17093140 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:54 proc_lsst.multi:146 [INFO] found job 17093142 in provider escience_a40 proc_lsst.multi INFO: found job 17093142 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bc759a6d3df22a1a7bf272ad53eeba5 with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/8e294298-b1a9-4c93-8d35-6cd985a20475_isrForCrosstalkSources_998108_40.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/8e294298-b1a9-4c93-8d35-6cd985a20475_isrForCrosstalkSources_998108_40.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 987f1b3ccb948c99e44ea63b0bc18e9f with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/9e098689-2431-40ba-bc5d-e98514fdbb4e_isrForCrosstalkSources_998211_18.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/9e098689-2431-40ba-bc5d-e98514fdbb4e_isrForCrosstalkSources_998211_18.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fa0a17f628e146a46f1b38064c3520e with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/5e45f162-54b7-42e2-b7b9-dc9de54a91fe_isrForCrosstalkSources_998106_33.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/5e45f162-54b7-42e2-b7b9-dc9de54a91fe_isrForCrosstalkSources_998106_33.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17fbdd1532b20e23b605984bc49b29f3 with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/0405daba-c5f4-4629-91cd-3bc077fd074f_isrForCrosstalkSources_998211_60.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/0405daba-c5f4-4629-91cd-3bc077fd074f_isrForCrosstalkSources_998211_60.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d44c28e884f766ea14978f3360de2e3 with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/c9124bf5-7869-4800-ab3a-c30b988a18f6_isrForCrosstalkSources_998108_29.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/c9124bf5-7869-4800-ab3a-c30b988a18f6_isrForCrosstalkSources_998108_29.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9431c9e3d6cbb7f6548505f4069757a with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/736e5649-8b6d-4daf-b44a-817750c50b22_isrForCrosstalkSources_998211_44.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/736e5649-8b6d-4daf-b44a-817750c50b22_isrForCrosstalkSources_998211_44.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d00711cfadfd124dc7c24a4b40864c7 with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/a821e10d-d905-4ea7-abcc-0e091dbbdb62_isrForCrosstalkSources_998212_14.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/a821e10d-d905-4ea7-abcc-0e091dbbdb62_isrForCrosstalkSources_998212_14.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 710c11135c8d3cb40c0e585917696a92 with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/739349c5-118c-4f9b-8410-273019e5939f_isrForCrosstalkSources_998211_21.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/739349c5-118c-4f9b-8410-273019e5939f_isrForCrosstalkSources_998211_21.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57858a26bba8b9f07f7fb0efa8173747 with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/9aaa9385-85fa-42b6-b978-432478a14acc_isrForCrosstalkSources_998107_14.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/9aaa9385-85fa-42b6-b978-432478a14acc_isrForCrosstalkSources_998107_14.stderr parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df3494e46386be418fc0f4619c39ac2a with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/fe0051eb-efec-4d10-b2c8-2b5c4f4cf7f7_isrForCrosstalkSources_998108_42.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/fe0051eb-efec-4d10-b2c8-2b5c4f4cf7f7_isrForCrosstalkSources_998108_42.stderr parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f7a95146196668b42d043481cd426d9 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/d0197f6d-d65e-4876-a16b-f73d0000e4e0_isrForCrosstalkSources_998108_34.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/d0197f6d-d65e-4876-a16b-f73d0000e4e0_isrForCrosstalkSources_998108_34.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5290962272b53531e9731bf08c2d8c67 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/1396534f-d8aa-4c9c-bf85-329d126a2be7_isrForCrosstalkSources_998211_40.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/1396534f-d8aa-4c9c-bf85-329d126a2be7_isrForCrosstalkSources_998211_40.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4e97170402f319ce24e71e420f09f34 with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/0c3c6df5-69e3-4cfd-94f1-64d87ab2d0d5_isrForCrosstalkSources_998211_26.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/0c3c6df5-69e3-4cfd-94f1-64d87ab2d0d5_isrForCrosstalkSources_998211_26.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 228ab7474024a6503e2699fbc1726098 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/240e0c3a-d4f4-4f1c-8964-57acf74e37af_isrForCrosstalkSources_998211_36.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/240e0c3a-d4f4-4f1c-8964-57acf74e37af_isrForCrosstalkSources_998211_36.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f099a69c640458a4912d67c847022504 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/e13c21f3-e94c-464e-9e89-bbd2d8bef640_isrForCrosstalkSources_998107_42.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/e13c21f3-e94c-464e-9e89-bbd2d8bef640_isrForCrosstalkSources_998107_42.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68d28dd051d23243639caf0cf27e1399 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/ee0f82c6-2a80-4462-86cc-325c0e534d92_isrForCrosstalkSources_998107_15.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/ee0f82c6-2a80-4462-86cc-325c0e534d92_isrForCrosstalkSources_998107_15.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae7f052d18eb9bb58508bb95950339cc with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/dc720944-adf2-4203-a00d-68c85cce107d_isrForCrosstalkSources_998211_12.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/dc720944-adf2-4203-a00d-68c85cce107d_isrForCrosstalkSources_998211_12.stderr parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e6fc8bffa740bc18200c9cbfb0af2c5 with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/8445c22c-996f-432b-8aec-2e91f090df51_isrForCrosstalkSources_998212_2.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/8445c22c-996f-432b-8aec-2e91f090df51_isrForCrosstalkSources_998212_2.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d34a242d716737794bf6eab3caf99f55 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/1f6c135f-5bb2-446e-af75-531df35c5532_isrForCrosstalkSources_998108_17.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/1f6c135f-5bb2-446e-af75-531df35c5532_isrForCrosstalkSources_998108_17.stderr parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f91987a2ebf3869b079392f56639954 with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/5d09b665-b87f-439a-ba25-0b2c4e063bf2_isrForCrosstalkSources_998210_31.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/5d09b665-b87f-439a-ba25-0b2c4e063bf2_isrForCrosstalkSources_998210_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:54 proc_lsst.multi:146 [INFO] found job 17093143 in provider escience_a40 proc_lsst.multi INFO: found job 17093143 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:55 proc_lsst.multi:146 [INFO] found job 17093145 in provider escience_a40 proc_lsst.multi INFO: found job 17093145 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:55 proc_lsst.multi:146 [INFO] found job 17093146 in provider escience_a40 proc_lsst.multi INFO: found job 17093146 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 212067eb3e2a330ddf90f7b8f119eb67 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/f7c6fe07-68e9-4aa2-9706-e3d5632ffc33_isrForCrosstalkSources_998108_33.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/f7c6fe07-68e9-4aa2-9706-e3d5632ffc33_isrForCrosstalkSources_998108_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:56 proc_lsst.multi:146 [INFO] found job 17093147 in provider escience_a40 proc_lsst.multi INFO: found job 17093147 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04388e3c2c337becf711d27501bb518f with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/37426948-d34c-4598-9ed8-3c3244e02cca_isrForCrosstalkSources_998210_59.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/37426948-d34c-4598-9ed8-3c3244e02cca_isrForCrosstalkSources_998210_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:47:56 proc_lsst.multi:146 [INFO] found job 17093149 in provider checkpoint proc_lsst.multi INFO: found job 17093149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:47:57 proc_lsst.multi:146 [INFO] found job 17093150 in provider checkpoint proc_lsst.multi INFO: found job 17093150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e23a50ae31ac6d27865e9126df5087e2 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/a2cf66dc-22e1-46d0-9a19-ba7fbdfb32c7_isrForCrosstalkSources_998210_28.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/a2cf66dc-22e1-46d0-9a19-ba7fbdfb32c7_isrForCrosstalkSources_998210_28.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b209aca12f04d9bab7a0a90d297f132a with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/fdaf5b46-b707-4b65-b97b-056168a3860c_isrForCrosstalkSources_998212_37.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/fdaf5b46-b707-4b65-b97b-056168a3860c_isrForCrosstalkSources_998212_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:47:57 proc_lsst.multi:146 [INFO] found job 17093151 in provider checkpoint proc_lsst.multi INFO: found job 17093151 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 598ea1e338023595ebdcd551199b44d8 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/32c1a5ff-405b-4220-9f81-b492482167ea_isrForCrosstalkSources_998106_11.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/32c1a5ff-405b-4220-9f81-b492482167ea_isrForCrosstalkSources_998106_11.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0ce7ccb71eb56661015a5721fb3fb9b with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/c31e2e4b-5fac-4a27-92d8-91c61ca7bfb9_isrForCrosstalkSources_998210_40.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/c31e2e4b-5fac-4a27-92d8-91c61ca7bfb9_isrForCrosstalkSources_998210_40.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 182afa13e1b29e77ed82e2c374bdfb9f with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/eabeb3a5-b28a-411f-9956-45aac436d7e1_isrForCrosstalkSources_998107_34.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/eabeb3a5-b28a-411f-9956-45aac436d7e1_isrForCrosstalkSources_998107_34.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48ce2107539428e165eff4b36f7434e3 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/51bc0899-2611-4ad2-8c2e-3b60f9d73a83_isrForCrosstalkSources_998106_17.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/51bc0899-2611-4ad2-8c2e-3b60f9d73a83_isrForCrosstalkSources_998106_17.stderr parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff45a362f88b2e684487b8722196fe62 with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/feb19a91-5677-4f6a-af49-37db96185d72_isrForCrosstalkSources_998211_17.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/feb19a91-5677-4f6a-af49-37db96185d72_isrForCrosstalkSources_998211_17.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 739d0fcda6500aacb7e9197624297f59 with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/00dde2fb-a720-41b3-a799-48c63c3bfbd7_isrForCrosstalkSources_998211_20.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/00dde2fb-a720-41b3-a799-48c63c3bfbd7_isrForCrosstalkSources_998211_20.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9542a8f00d25610a48723229ae3cdde1 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/08467930-8d13-4950-b1f2-b523d10e5ef2_isrForCrosstalkSources_998211_41.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/08467930-8d13-4950-b1f2-b523d10e5ef2_isrForCrosstalkSources_998211_41.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 943bcb219d9bc088b2461b4d09ca778d with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/3c4b2933-f7ea-46b4-819e-77cd0ec2d373_isrForCrosstalkSources_998107_12.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/3c4b2933-f7ea-46b4-819e-77cd0ec2d373_isrForCrosstalkSources_998107_12.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46937c8feae7676e52a95c347931b774 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/a9125b79-5edc-46b4-be38-4ce143685b26_isrForCrosstalkSources_998107_38.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/a9125b79-5edc-46b4-be38-4ce143685b26_isrForCrosstalkSources_998107_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:47:59 proc_lsst.multi:146 [INFO] found job 17093153 in provider checkpoint proc_lsst.multi INFO: found job 17093153 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b3ae7a03b5a787beb012658539901b1 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/6db42cf2-30a9-4419-ac60-aacfaa9ae88d_isrForCrosstalkSources_998106_35.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/6db42cf2-30a9-4419-ac60-aacfaa9ae88d_isrForCrosstalkSources_998106_35.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7db6ba69437e17ed2ca0bc99e3c92294 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/432e24a4-a756-4d10-a6f4-d2d01c3ff85f_isrForCrosstalkSources_998107_49.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/432e24a4-a756-4d10-a6f4-d2d01c3ff85f_isrForCrosstalkSources_998107_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:00 proc_lsst.multi:146 [INFO] found job 17093154 in provider checkpoint proc_lsst.multi INFO: found job 17093154 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34db750c55d985dcc787fe2e98e0c8fc with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/123ece1c-5ce9-42cd-8dd5-9f76d3baba8f_isrForCrosstalkSources_998108_30.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/123ece1c-5ce9-42cd-8dd5-9f76d3baba8f_isrForCrosstalkSources_998108_30.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95b14a1391a54374cf6ea1ac126429f5 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/f72e96ff-db9d-4a93-ab0d-c4be135d9c1a_isrForCrosstalkSources_998212_58.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/f72e96ff-db9d-4a93-ab0d-c4be135d9c1a_isrForCrosstalkSources_998212_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:01 proc_lsst.multi:146 [INFO] found job 17093155 in provider checkpoint proc_lsst.multi INFO: found job 17093155 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d750e7438e5ff52f24df2c2bc86800b1 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/e6fde39c-fedd-45ef-83f3-8cd39042c96d_isrForCrosstalkSources_998108_2.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/e6fde39c-fedd-45ef-83f3-8cd39042c96d_isrForCrosstalkSources_998108_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:01 proc_lsst.multi:146 [INFO] found job 17093157 in provider checkpoint proc_lsst.multi INFO: found job 17093157 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a63d3bc0843b88a1d66a079a31c2019 with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/f304118b-ab79-458e-8a4a-e5b52bd5ad0f_isrForCrosstalkSources_998210_32.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/f304118b-ab79-458e-8a4a-e5b52bd5ad0f_isrForCrosstalkSources_998210_32.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c47bd144017353fb83578736a510bc1 with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/affe2d51-1963-4da2-8404-a64028804c68_isrForCrosstalkSources_998108_9.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/affe2d51-1963-4da2-8404-a64028804c68_isrForCrosstalkSources_998108_9.stderr parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry effb9d6b50ce547400349b3317fc64d2 with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/7c5f999d-4d9c-427b-9b59-cb6c3cf58616_isrForCrosstalkSources_998108_7.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/7c5f999d-4d9c-427b-9b59-cb6c3cf58616_isrForCrosstalkSources_998108_7.stderr parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ee2a7a0598170ca63a1c99f51424796 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/bc40e7c6-5a67-4cd3-83f2-48929377db30_isrForCrosstalkSources_998108_21.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/bc40e7c6-5a67-4cd3-83f2-48929377db30_isrForCrosstalkSources_998108_21.stderr parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6503b46178d48405483274604c2be45c with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/e0da7857-942a-4d3e-8bba-c1bf20444fbf_isrForCrosstalkSources_998108_10.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/e0da7857-942a-4d3e-8bba-c1bf20444fbf_isrForCrosstalkSources_998108_10.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 251595125f7a563e92d656ca9b8bc11f with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/9ad2a95c-8330-44b1-af42-c11b0d6b83fc_isrForCrosstalkSources_998108_24.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/9ad2a95c-8330-44b1-af42-c11b0d6b83fc_isrForCrosstalkSources_998108_24.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f7c3128908f8a72f83437b5a532f552 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/06d3440b-29d2-41c3-b6a8-d00c22a3e3eb_isrForCrosstalkSources_998108_3.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/06d3440b-29d2-41c3-b6a8-d00c22a3e3eb_isrForCrosstalkSources_998108_3.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82ab308e301a994fb96dcf04b1b6097c with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/7d5f5af2-ce71-4888-9240-0c3e54722dd6_isrForCrosstalkSources_998210_62.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/7d5f5af2-ce71-4888-9240-0c3e54722dd6_isrForCrosstalkSources_998210_62.stderr parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dc1f4f22e86899342e6b5fa9e1acbb6 with result from task 71 parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:01 proc_lsst.multi:146 [INFO] found job 17093158 in provider checkpoint proc_lsst.multi INFO: found job 17093158 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/9c34cc71-1a6f-4755-8e12-8914d3ae06da_isrForCrosstalkSources_998210_37.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/9c34cc71-1a6f-4755-8e12-8914d3ae06da_isrForCrosstalkSources_998210_37.stderr parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 075c3200cceaeae485550159c5f3b83e with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/1c84e32b-b9f1-4814-8fdc-dafd4fff9944_isrForCrosstalkSources_998210_61.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/1c84e32b-b9f1-4814-8fdc-dafd4fff9944_isrForCrosstalkSources_998210_61.stderr parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ed2b9d1b6f521a388be371a0d4e2491 with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/3daec035-0a91-4e21-87f2-d5f99457c8fd_isrForCrosstalkSources_998108_31.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/3daec035-0a91-4e21-87f2-d5f99457c8fd_isrForCrosstalkSources_998108_31.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2f17ea90509ea557555a5da5700c7ca with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/91bf3645-178e-47d0-9344-fcddeaef5376_isrForCrosstalkSources_998211_9.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/91bf3645-178e-47d0-9344-fcddeaef5376_isrForCrosstalkSources_998211_9.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec7f677d0f5f2ce4e976d60b1e71b476 with result from task 75 parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:02 proc_lsst.multi:146 [INFO] found job 17093159 in provider checkpoint proc_lsst.multi INFO: found job 17093159 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/f337e158-b091-4d37-9990-7da49cfbafcf_isrForCrosstalkSources_998108_52.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/f337e158-b091-4d37-9990-7da49cfbafcf_isrForCrosstalkSources_998108_52.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55413e969c45b4bc2b4ff67fad70eed3 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/39d6521c-c545-4c4e-a1a6-febf76f5b744_isrForCrosstalkSources_998108_54.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/39d6521c-c545-4c4e-a1a6-febf76f5b744_isrForCrosstalkSources_998108_54.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca39e629f408138cf15acbc1a1d2ce84 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/f520da5e-f86e-4061-b873-48650dbf26da_isrForCrosstalkSources_998108_57.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/f520da5e-f86e-4061-b873-48650dbf26da_isrForCrosstalkSources_998108_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:02 proc_lsst.multi:146 [INFO] found job 17093160 in provider checkpoint proc_lsst.multi INFO: found job 17093160 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 788470ab9c5f29533fe372906c833ee1 with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/debf6e4d-b555-4ec7-ab05-f14219b243c5_isrForCrosstalkSources_998107_43.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/debf6e4d-b555-4ec7-ab05-f14219b243c5_isrForCrosstalkSources_998107_43.stderr parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 534ce214e567ea92f8425f7fcdfe60f1 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/57a9cade-247a-4418-a309-f4c02379e11d_isrForCrosstalkSources_998107_50.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/57a9cade-247a-4418-a309-f4c02379e11d_isrForCrosstalkSources_998107_50.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c27439af42ec9dc901b2e9d274b35b7 with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/87470ff5-320b-4af3-95da-144469ef3a3a_isrForCrosstalkSources_998107_29.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/87470ff5-320b-4af3-95da-144469ef3a3a_isrForCrosstalkSources_998107_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:03 proc_lsst.multi:146 [INFO] found job 17093161 in provider checkpoint proc_lsst.multi INFO: found job 17093161 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:03 proc_lsst.multi:146 [INFO] found job 17093162 in provider checkpoint proc_lsst.multi INFO: found job 17093162 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:03 proc_lsst.multi:146 [INFO] found job 17093163 in provider checkpoint proc_lsst.multi INFO: found job 17093163 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:03 proc_lsst.multi:146 [INFO] found job 17093164 in provider checkpoint proc_lsst.multi INFO: found job 17093164 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 113308a31c55166e42cd170f27943c07 with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/422903b9-71ed-4db6-83dd-762e5d8dec71_isrForCrosstalkSources_998106_28.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/422903b9-71ed-4db6-83dd-762e5d8dec71_isrForCrosstalkSources_998106_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:04 proc_lsst.multi:146 [INFO] found job 17093165 in provider checkpoint proc_lsst.multi INFO: found job 17093165 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:04 proc_lsst.multi:146 [INFO] found job 17093166 in provider checkpoint proc_lsst.multi INFO: found job 17093166 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf03ec8ed1bb3e6ad82a801a7776373c with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/585a094e-0fc5-404c-a373-d188697947d9_isrForCrosstalkSources_998210_9.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/585a094e-0fc5-404c-a373-d188697947d9_isrForCrosstalkSources_998210_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:05 proc_lsst.multi:146 [INFO] found job 17093167 in provider checkpoint proc_lsst.multi INFO: found job 17093167 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e78dff2a4ee70f91f5ebac5d93ce086b with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/17e25ac4-9de9-4fb3-a8d3-b834dd3604d8_isrForCrosstalkSources_998210_33.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/17e25ac4-9de9-4fb3-a8d3-b834dd3604d8_isrForCrosstalkSources_998210_33.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12c07fa0383e9ececec9366342e81ce7 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/d7addd36-ae0a-444d-b21b-f2b064a00fa6_isrForCrosstalkSources_998210_47.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/d7addd36-ae0a-444d-b21b-f2b064a00fa6_isrForCrosstalkSources_998210_47.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15d76a7fc7065e5a0e6169f9fd42ae34 with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/ff92261f-871f-4d48-9e17-4ace669467d1_isrForCrosstalkSources_998210_41.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/ff92261f-871f-4d48-9e17-4ace669467d1_isrForCrosstalkSources_998210_41.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75e1392eba1f7befa003b982d274a1f6 with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/29c55c5b-1650-4e24-b945-ebe0351a2920_isrForCrosstalkSources_998108_1.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/29c55c5b-1650-4e24-b945-ebe0351a2920_isrForCrosstalkSources_998108_1.stderr parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cea1bbaab454ef1bd2cdde478d576c1 with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/ee88bf76-94b1-485f-944b-57b15eef8f23_isrForCrosstalkSources_998108_11.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/ee88bf76-94b1-485f-944b-57b15eef8f23_isrForCrosstalkSources_998108_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:05 proc_lsst.multi:146 [INFO] found job 17093168 in provider checkpoint proc_lsst.multi INFO: found job 17093168 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c3c38da89039c3923883942055c77bb with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/aa5f56a1-4c0d-4af5-a7ee-39e344cc02aa_isrForCrosstalkSources_998210_43.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/aa5f56a1-4c0d-4af5-a7ee-39e344cc02aa_isrForCrosstalkSources_998210_43.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03844d7e47d4f24d1fe798d6c3b6def2 with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/2a8ab7d8-1fd1-44f0-95a3-956be18dcaec_isrForCrosstalkSources_998210_44.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/2a8ab7d8-1fd1-44f0-95a3-956be18dcaec_isrForCrosstalkSources_998210_44.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f978cafa84ddaa8d644bd1abe0258163 with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/a564c785-470e-4b78-80e5-a6fa7b789e70_isrForCrosstalkSources_998108_5.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/a564c785-470e-4b78-80e5-a6fa7b789e70_isrForCrosstalkSources_998108_5.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a599e18d86d8493e7a1fef5d35742296 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/118fd9bc-8331-43f0-9c7d-8017bd9a6308_isrForCrosstalkSources_998107_18.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/118fd9bc-8331-43f0-9c7d-8017bd9a6308_isrForCrosstalkSources_998107_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:06 proc_lsst.multi:146 [INFO] found job 17093169 in provider checkpoint proc_lsst.multi INFO: found job 17093169 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a672ee720c55832390a2763852b3d326 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/214320ca-92ca-4a81-9977-70b4cb27564f_isrForCrosstalkSources_998212_51.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/214320ca-92ca-4a81-9977-70b4cb27564f_isrForCrosstalkSources_998212_51.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 337c152739abaf775a97d32054730e0d with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/9fc78d1c-5d2f-4035-8ed2-d967b4e8a081_isrForCrosstalkSources_998108_60.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/9fc78d1c-5d2f-4035-8ed2-d967b4e8a081_isrForCrosstalkSources_998108_60.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d18ed09f55e668d67d9c34ee0e0dc164 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/fce460bc-9311-4b23-86fd-fff8466ac91a_isrForCrosstalkSources_998107_10.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/fce460bc-9311-4b23-86fd-fff8466ac91a_isrForCrosstalkSources_998107_10.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33116c3f247609e0d4a54c3cec7eb622 with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/19b46b69-6563-4657-a5ad-6324f06f87c9_isrForCrosstalkSources_998107_23.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/19b46b69-6563-4657-a5ad-6324f06f87c9_isrForCrosstalkSources_998107_23.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8eac15ea74dde0deeb4d722a1b4f8701 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/9f04b63f-3c84-4a58-b608-3fec2b2caf8b_isrForCrosstalkSources_998210_51.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/9f04b63f-3c84-4a58-b608-3fec2b2caf8b_isrForCrosstalkSources_998210_51.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3179575c5b9d10b4974a93959c9c859 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/522a237e-ed78-4a64-9916-7e66fb65b732_isrForCrosstalkSources_998211_5.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/522a237e-ed78-4a64-9916-7e66fb65b732_isrForCrosstalkSources_998211_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:06 proc_lsst.multi:146 [INFO] found job 17093170 in provider checkpoint proc_lsst.multi INFO: found job 17093170 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6eda90e67851fe9cb5a7e83a67f9d07 with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/0cf24121-70bb-4fde-bdcf-400513ef8397_isrForCrosstalkSources_998108_47.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/0cf24121-70bb-4fde-bdcf-400513ef8397_isrForCrosstalkSources_998108_47.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f86c14c16a8474055c3bc619facea84c with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/6c608334-41ec-4e7f-a0a1-d5d41318ae67_isrForCrosstalkSources_998107_44.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/6c608334-41ec-4e7f-a0a1-d5d41318ae67_isrForCrosstalkSources_998107_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:07 proc_lsst.multi:146 [INFO] found job 17093171 in provider checkpoint proc_lsst.multi INFO: found job 17093171 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5787e21d6ea2dd20970b904117b246a1 with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/fced1cea-b533-41b7-afa3-896d703e2471_isrForCrosstalkSources_998212_35.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/fced1cea-b533-41b7-afa3-896d703e2471_isrForCrosstalkSources_998212_35.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f8325869a04b2ec5d656f908d9c6a88 with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/36570a24-2920-412b-968d-5fc57645b155_isrForCrosstalkSources_998108_13.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/36570a24-2920-412b-968d-5fc57645b155_isrForCrosstalkSources_998108_13.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc1e7515d58a61a039ba3bce5046e989 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/4b270735-285c-4d20-93f3-101a6abd99a0_isrForCrosstalkSources_998212_11.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/4b270735-285c-4d20-93f3-101a6abd99a0_isrForCrosstalkSources_998212_11.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8221d4eaffdc3570ebf766a9f5d39ba with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/36cbee7e-db74-46ea-b13e-3e1e64de8344_isrForCrosstalkSources_998212_43.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/36cbee7e-db74-46ea-b13e-3e1e64de8344_isrForCrosstalkSources_998212_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:07 proc_lsst.multi:146 [INFO] found job 17093172 in provider checkpoint proc_lsst.multi INFO: found job 17093172 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c6c51d288b278e74b706ac5a3e0bdda with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/83279349-983f-4a47-808e-dc084686d063_isrForCrosstalkSources_998210_56.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/83279349-983f-4a47-808e-dc084686d063_isrForCrosstalkSources_998210_56.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 545c2c306129447b07070f87b1ca423e with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/409a0357-3222-41fc-be24-6e2bcdd9cabb_isrForCrosstalkSources_998210_4.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/409a0357-3222-41fc-be24-6e2bcdd9cabb_isrForCrosstalkSources_998210_4.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 092784642a6133a0d2c58b08fee06c45 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/1a110039-2602-47ea-9e91-9331f3c2630d_isrForCrosstalkSources_998106_1.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/1a110039-2602-47ea-9e91-9331f3c2630d_isrForCrosstalkSources_998106_1.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ddf68a7adb2e948db77fa2840c49f81 with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/ad533053-e53f-4c11-a420-2b26a5d701b2_isrForCrosstalkSources_998106_58.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/ad533053-e53f-4c11-a420-2b26a5d701b2_isrForCrosstalkSources_998106_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:08 proc_lsst.multi:146 [INFO] found job 17093173 in provider checkpoint proc_lsst.multi INFO: found job 17093173 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c23fb274b3c35a01e1acea4f3705ed5 with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/ab2e173d-79ea-402c-9582-39d2d012f14d_isrForCrosstalkSources_998212_48.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/ab2e173d-79ea-402c-9582-39d2d012f14d_isrForCrosstalkSources_998212_48.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31fca1102aa37a8bb5104de2ae9b5533 with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/5eed6233-ae15-4bf3-8c48-5db75b66d2de_isrForCrosstalkSources_998108_48.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/5eed6233-ae15-4bf3-8c48-5db75b66d2de_isrForCrosstalkSources_998108_48.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 260a52764ae8a60de01c0e2816077f2b with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/89014e12-3857-4c8d-a9c8-6a1fe33bfe54_isrForCrosstalkSources_998210_45.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/89014e12-3857-4c8d-a9c8-6a1fe33bfe54_isrForCrosstalkSources_998210_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:08 proc_lsst.multi:146 [INFO] found job 17093174 in provider checkpoint proc_lsst.multi INFO: found job 17093174 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcc6ab635ddde15b933f929c94a1b554 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/8d4c9a89-3587-49f4-a29f-9e354571cb12_isrForCrosstalkSources_998107_55.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/8d4c9a89-3587-49f4-a29f-9e354571cb12_isrForCrosstalkSources_998107_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:09 proc_lsst.multi:146 [INFO] found job 17093175 in provider checkpoint proc_lsst.multi INFO: found job 17093175 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38b64487d0cd0380993cba1ff2e5d360 with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/dc652169-c5b1-40a5-b1d3-9cf1b8f8053f_isrForCrosstalkSources_998211_23.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/dc652169-c5b1-40a5-b1d3-9cf1b8f8053f_isrForCrosstalkSources_998211_23.stderr parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 870515d3c291fe7c4b09afd04ccaac7f with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/50db07e7-2f54-4424-bd39-afc66f8089ab_isrForCrosstalkSources_998212_29.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/50db07e7-2f54-4424-bd39-afc66f8089ab_isrForCrosstalkSources_998212_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:09 proc_lsst.multi:146 [INFO] found job 17093176 in provider checkpoint proc_lsst.multi INFO: found job 17093176 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a690f0b91c3cb4a95e2be53a9047cba with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/d1ee50e1-b5fd-4c50-afd4-642c22d94fec_isrForCrosstalkSources_998210_36.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/d1ee50e1-b5fd-4c50-afd4-642c22d94fec_isrForCrosstalkSources_998210_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:10 proc_lsst.multi:146 [INFO] found job 17093177 in provider checkpoint proc_lsst.multi INFO: found job 17093177 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 902289b537c08173b75cb92e964024ab with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/0fe8964a-5225-4159-8582-b4573bc3d7fe_isrForCrosstalkSources_998211_24.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/0fe8964a-5225-4159-8582-b4573bc3d7fe_isrForCrosstalkSources_998211_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:10 proc_lsst.multi:146 [INFO] found job 17093178 in provider checkpoint proc_lsst.multi INFO: found job 17093178 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:11 proc_lsst.multi:146 [INFO] found job 17093179 in provider checkpoint proc_lsst.multi INFO: found job 17093179 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6eca4e9283eef14c41552aa39add9dcb with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/760ef7a0-eb4a-4b4f-a407-7c399b09102b_isrForCrosstalkSources_998107_22.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/760ef7a0-eb4a-4b4f-a407-7c399b09102b_isrForCrosstalkSources_998107_22.stderr parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed66422a81d15f3de794b93f5c899da0 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/6382dd28-b59e-4ad1-badd-19f1e2692445_isrForCrosstalkSources_998106_16.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/6382dd28-b59e-4ad1-badd-19f1e2692445_isrForCrosstalkSources_998106_16.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ab692b8a6646bb94d5e8d886496a40f with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/e5dc846d-8bcb-4621-a396-c53d3e89d197_isrForCrosstalkSources_998211_32.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/e5dc846d-8bcb-4621-a396-c53d3e89d197_isrForCrosstalkSources_998211_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:11 proc_lsst.multi:146 [INFO] found job 17093180 in provider checkpoint proc_lsst.multi INFO: found job 17093180 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4928207018a826351faab84ec5074c64 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/9cf3e5f7-9c5e-4d0f-9c28-8099debaca49_isrForCrosstalkSources_998211_16.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/9cf3e5f7-9c5e-4d0f-9c28-8099debaca49_isrForCrosstalkSources_998211_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:12 proc_lsst.multi:146 [INFO] found job 17093181 in provider checkpoint proc_lsst.multi INFO: found job 17093181 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:12 proc_lsst.multi:146 [INFO] found job 17093182 in provider checkpoint proc_lsst.multi INFO: found job 17093182 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:13 proc_lsst.multi:146 [INFO] found job 17093183 in provider checkpoint proc_lsst.multi INFO: found job 17093183 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:13 proc_lsst.multi:146 [INFO] found job 17093184 in provider checkpoint proc_lsst.multi INFO: found job 17093184 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:13 proc_lsst.multi:146 [INFO] found job 17093185 in provider checkpoint proc_lsst.multi INFO: found job 17093185 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:13 proc_lsst.multi:146 [INFO] found job 17093186 in provider checkpoint proc_lsst.multi INFO: found job 17093186 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING 2024-03-15 22:48:14 proc_lsst.multi:146 [INFO] found job 17093187 in provider checkpoint proc_lsst.multi INFO: found job 17093187 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093149 PD 17093150 PD 17093151 PD 17093153 PD 17093154 PD 17093155 PD 17093157 PD 17093158 PD 17093159 PD 17093160 PD 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 245 parsl.dataflow.strategy DEBUG: Executor multi has 245 active tasks, 56/37 running/pending blocks, and 57 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22773982342992 2024-03-15 22:48:14 proc_lsst.multi:146 [INFO] found job 36385 in provider local proc_lsst.multi INFO: found job 36385 in provider local 2024-03-15 22:48:14 proc_lsst.multi:146 [INFO] found job 17093085 in provider astro proc_lsst.multi INFO: found job 17093085 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:15 proc_lsst.multi:146 [INFO] found job 17093086 in provider astro proc_lsst.multi INFO: found job 17093086 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40c49403e1428eb29214845f0dd27f7e with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/b561cad7-1d7b-42c0-a40a-4ad9ca734561_isrForCrosstalkSources_998210_42.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/b561cad7-1d7b-42c0-a40a-4ad9ca734561_isrForCrosstalkSources_998210_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:15 proc_lsst.multi:146 [INFO] found job 17093087 in provider astro proc_lsst.multi INFO: found job 17093087 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:16 proc_lsst.multi:146 [INFO] found job 17093088 in provider astro proc_lsst.multi INFO: found job 17093088 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:16 proc_lsst.multi:146 [INFO] found job 17093089 in provider astro proc_lsst.multi INFO: found job 17093089 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7aa321a1d7133922108584569b72eff1 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/479bbbd9-c0da-498d-85bf-ac5b4c911753_isrForCrosstalkSources_998212_49.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/479bbbd9-c0da-498d-85bf-ac5b4c911753_isrForCrosstalkSources_998212_49.stderr parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3097e0224af802e2a25396e2632264ef with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/170d62af-2c83-44be-a1de-c845a449f5fc_isrForCrosstalkSources_998211_50.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/170d62af-2c83-44be-a1de-c845a449f5fc_isrForCrosstalkSources_998211_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:17 proc_lsst.multi:146 [INFO] found job 17093090 in provider astro proc_lsst.multi INFO: found job 17093090 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7dbfdb83170e5bcb515249678243c2a3 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/73a561e4-5d10-4835-81a0-006a563f7a2f_isrForCrosstalkSources_998107_53.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/73a561e4-5d10-4835-81a0-006a563f7a2f_isrForCrosstalkSources_998107_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:17 proc_lsst.multi:146 [INFO] found job 17093091 in provider astro proc_lsst.multi INFO: found job 17093091 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0f7e762eddce492fb4f02df8d9861aa with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/c1c9d76d-66ca-454e-97cc-185bd2b45152_isrForCrosstalkSources_998108_14.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/c1c9d76d-66ca-454e-97cc-185bd2b45152_isrForCrosstalkSources_998108_14.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc5d6a3370bb3a8e513138544666f5cb with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/b713906a-5d3d-45a4-946e-eb1f23400edb_isrForCrosstalkSources_998210_26.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/b713906a-5d3d-45a4-946e-eb1f23400edb_isrForCrosstalkSources_998210_26.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e233fc641ac19360f02fe5fddffb7db7 with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/c3a097f5-306a-4053-bc10-204ecbd9d95e_isrForCrosstalkSources_998210_5.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/c3a097f5-306a-4053-bc10-204ecbd9d95e_isrForCrosstalkSources_998210_5.stderr parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e60fba1b149cbb26ec4ecc342079d49a with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/8124448e-e27a-433c-8d9e-0323561a7a1c_isrForCrosstalkSources_998210_11.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/8124448e-e27a-433c-8d9e-0323561a7a1c_isrForCrosstalkSources_998210_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:18 proc_lsst.multi:146 [INFO] found job 17093092 in provider astro proc_lsst.multi INFO: found job 17093092 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f109bb5df13f168ed2fe910250aeba09 with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/57ca2f81-0663-4abb-b803-e57883758640_isrForCrosstalkSources_998210_13.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/57ca2f81-0663-4abb-b803-e57883758640_isrForCrosstalkSources_998210_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:18 proc_lsst.multi:146 [INFO] found job 17093093 in provider astro proc_lsst.multi INFO: found job 17093093 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39b7800ad14a7e7d909638dadd4ff36b with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/7bff279f-0038-49e7-a4e8-9da13cbd5ccd_isrForCrosstalkSources_998106_12.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/7bff279f-0038-49e7-a4e8-9da13cbd5ccd_isrForCrosstalkSources_998106_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:19 proc_lsst.multi:146 [INFO] found job 17093094 in provider astro proc_lsst.multi INFO: found job 17093094 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a39f823e9acce935e2c238fc104cb50e with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/e1e9cdda-ebb3-4d86-b627-5f1bf781af01_isrForCrosstalkSources_998211_22.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/e1e9cdda-ebb3-4d86-b627-5f1bf781af01_isrForCrosstalkSources_998211_22.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6988287e6d25aecfe84d47372ec07cb with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/de7e65e4-2686-4760-8062-8d91d8587082_isrForCrosstalkSources_998210_20.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/de7e65e4-2686-4760-8062-8d91d8587082_isrForCrosstalkSources_998210_20.stderr parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e290f7a1f02dd040c7ca12bafb86941c with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/e170dc0a-2195-42dc-a4d5-c18283fb7cad_isrForCrosstalkSources_998211_33.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/e170dc0a-2195-42dc-a4d5-c18283fb7cad_isrForCrosstalkSources_998211_33.stderr parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76480146e2c5363767aee2d67913f1ce with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/86639e85-a782-41d4-bbd3-1544aace3d34_isrForCrosstalkSources_998211_45.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/86639e85-a782-41d4-bbd3-1544aace3d34_isrForCrosstalkSources_998211_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:19 proc_lsst.multi:146 [INFO] found job 17093095 in provider astro proc_lsst.multi INFO: found job 17093095 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b99ef7420885c0312318c53d9dae2fc with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/5942b0c1-9eb8-4a29-a390-b87cd92bd293_isrForCrosstalkSources_998106_37.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/5942b0c1-9eb8-4a29-a390-b87cd92bd293_isrForCrosstalkSources_998106_37.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8058bc5505dc0f20b954614e00cb7283 with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/bf016a6c-2280-45ab-bb21-0ea7c0815dcc_isrForCrosstalkSources_998211_47.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/bf016a6c-2280-45ab-bb21-0ea7c0815dcc_isrForCrosstalkSources_998211_47.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4297fcfd7052c539090d0ce4acf9839b with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/21bbd208-b4ab-46d3-8556-6bca47ba0525_isrForCrosstalkSources_998108_32.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/21bbd208-b4ab-46d3-8556-6bca47ba0525_isrForCrosstalkSources_998108_32.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1b0e99cd4455fc98e985899852d97a5 with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/d711bc0a-b597-4e84-b8b1-f18ae0ee772a_isrForCrosstalkSources_998210_21.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/d711bc0a-b597-4e84-b8b1-f18ae0ee772a_isrForCrosstalkSources_998210_21.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de6fcf4abdafe36a93c6e37e1820b245 with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/c4b5cd46-1c40-4e00-905d-fce322bdbb5f_isrForCrosstalkSources_998210_12.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/c4b5cd46-1c40-4e00-905d-fce322bdbb5f_isrForCrosstalkSources_998210_12.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ce07465843da1bf00cca98857db9bf1 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/0a1441f9-6276-4b87-b0c1-45e7a5f79730_isrForCrosstalkSources_998106_8.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/0a1441f9-6276-4b87-b0c1-45e7a5f79730_isrForCrosstalkSources_998106_8.stderr parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb08c17b5326f3d78c9c2b2fdb551e37 with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/2365cc45-d001-4dbb-a980-4a7999c47d8f_isrForCrosstalkSources_998106_22.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/2365cc45-d001-4dbb-a980-4a7999c47d8f_isrForCrosstalkSources_998106_22.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9194cfcaa1c519105b254b6fa4b42368 with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/07649b61-4df9-4b4f-8c81-a08a82d7e934_isrForCrosstalkSources_998106_39.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/07649b61-4df9-4b4f-8c81-a08a82d7e934_isrForCrosstalkSources_998106_39.stderr parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d5fb12f7ac3e23125b99677bc41e9af with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/d28b95cc-94f4-4e3d-bf45-15f4965cb187_isrForCrosstalkSources_998212_13.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/d28b95cc-94f4-4e3d-bf45-15f4965cb187_isrForCrosstalkSources_998212_13.stderr parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4924b883ec8dd5d3b35d95243079752 with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/1681163f-aa34-45d8-b978-7b1b2c9e5cd8_isrForCrosstalkSources_998210_60.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/1681163f-aa34-45d8-b978-7b1b2c9e5cd8_isrForCrosstalkSources_998210_60.stderr parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f215e744da1e14db06ab33dc2dcca6e5 with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/895e3f75-99bd-4af1-80e5-8e57cf805422_isrForCrosstalkSources_998211_1.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/895e3f75-99bd-4af1-80e5-8e57cf805422_isrForCrosstalkSources_998211_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:19 proc_lsst.multi:146 [INFO] found job 17093096 in provider astro proc_lsst.multi INFO: found job 17093096 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7a5f8d81dea258c59edffb9e19671a7 with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/25661b1e-64a5-4037-9f28-6652328af202_isrForCrosstalkSources_998108_22.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/25661b1e-64a5-4037-9f28-6652328af202_isrForCrosstalkSources_998108_22.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c1d4508eaa628de1238ba14b11c8c15 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/518e347f-eb09-4005-b46c-49f94399b3d8_isrForCrosstalkSources_998106_10.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/518e347f-eb09-4005-b46c-49f94399b3d8_isrForCrosstalkSources_998106_10.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26143a4ecf27cb9887945e9935087138 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/c20b6b04-0749-486f-8dcf-2ae4db78a7d0_isrForCrosstalkSources_998211_48.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/c20b6b04-0749-486f-8dcf-2ae4db78a7d0_isrForCrosstalkSources_998211_48.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a5a4571c012edb36556ebaf5315f674 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/54860232-d4fc-484c-90a6-b5857084f097_isrForCrosstalkSources_998212_25.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/54860232-d4fc-484c-90a6-b5857084f097_isrForCrosstalkSources_998212_25.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b31ad07f7627094dc99138186d493dab with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/aabe4cae-e4a3-4fb4-b2c6-9440b3fa155b_isrForCrosstalkSources_998107_32.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/aabe4cae-e4a3-4fb4-b2c6-9440b3fa155b_isrForCrosstalkSources_998107_32.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd56d3ecba7c02edec8aaa5cea898d70 with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/69ce9ea8-6dbd-468e-8e83-dab8191b768c_isrForCrosstalkSources_998107_19.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/69ce9ea8-6dbd-468e-8e83-dab8191b768c_isrForCrosstalkSources_998107_19.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e529c5df18e307d62f494507b179f6c with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/d9b6d7e2-a1f2-431c-b9fa-e0176fa341d5_isrForCrosstalkSources_998108_25.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/d9b6d7e2-a1f2-431c-b9fa-e0176fa341d5_isrForCrosstalkSources_998108_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:20 proc_lsst.multi:146 [INFO] found job 17093097 in provider astro proc_lsst.multi INFO: found job 17093097 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c68ea77c1d13d1c44b002d6e25b3f707 with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/39a34ee5-06a7-4257-b4cc-e580b3d65882_isrForCrosstalkSources_998106_59.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/39a34ee5-06a7-4257-b4cc-e580b3d65882_isrForCrosstalkSources_998106_59.stderr parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 911490187e713c017b88670301a79b5e with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/a6f91537-aa1c-4f79-85c4-c374c566804d_isrForCrosstalkSources_998211_4.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/a6f91537-aa1c-4f79-85c4-c374c566804d_isrForCrosstalkSources_998211_4.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 207689f4fcb26abebd62921dfd8cfb3e with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/8b4d7144-d9ff-442d-9b10-30314082e196_isrForCrosstalkSources_998210_34.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/8b4d7144-d9ff-442d-9b10-30314082e196_isrForCrosstalkSources_998210_34.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e71eb3484583b0555b73b7eb94c872e2 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/91c8d1e6-abcb-4b2f-9a81-75791fe39154_isrForCrosstalkSources_998212_39.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/91c8d1e6-abcb-4b2f-9a81-75791fe39154_isrForCrosstalkSources_998212_39.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c2d4969d9cb4c81be3ea47335e36b9f with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/3a04beef-dfec-42ff-8a90-9a389982db0b_isrForCrosstalkSources_998212_46.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/3a04beef-dfec-42ff-8a90-9a389982db0b_isrForCrosstalkSources_998212_46.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5d4fed304c5c560aca24f68c9044829 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/7266f033-ec69-424b-9808-601ffabb3882_isrForCrosstalkSources_998107_11.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/7266f033-ec69-424b-9808-601ffabb3882_isrForCrosstalkSources_998107_11.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54ab96973306751224eb171cf1efcba9 with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/21ca6938-da55-4469-814c-652e9d97a7f9_isrForCrosstalkSources_998108_4.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/21ca6938-da55-4469-814c-652e9d97a7f9_isrForCrosstalkSources_998108_4.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9fec743011c49f8ed7c52e6ada08ce9 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/4e770f1a-16c0-4834-9776-79e1e749b941_isrForCrosstalkSources_998108_56.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/4e770f1a-16c0-4834-9776-79e1e749b941_isrForCrosstalkSources_998108_56.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbf5c47741c3f1331fcd1911468832e1 with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/172dafc9-b240-4038-8c39-f833947bb969_isrForCrosstalkSources_998211_10.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/172dafc9-b240-4038-8c39-f833947bb969_isrForCrosstalkSources_998211_10.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e8f73163ba2f4addbb6080bd759303d with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/46b81430-1a16-423f-a244-b3ddd199a103_isrForCrosstalkSources_998211_28.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/46b81430-1a16-423f-a244-b3ddd199a103_isrForCrosstalkSources_998211_28.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0be3ca8275a14829241f22654a0b26c2 with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/e61fdf29-faa3-40bf-aa75-6d7a373e2cfb_isrForCrosstalkSources_998107_3.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/e61fdf29-faa3-40bf-aa75-6d7a373e2cfb_isrForCrosstalkSources_998107_3.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd5741f9260c287e1868137c0caf3b6c with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/e373348b-6b49-4232-8936-24f50c3dc14a_isrForCrosstalkSources_998211_42.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/e373348b-6b49-4232-8936-24f50c3dc14a_isrForCrosstalkSources_998211_42.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82540461e3639ee141ebcf1ce6f02034 with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/78b12a6b-df12-43cd-abc7-b0962b957ce2_isrForCrosstalkSources_998107_54.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/78b12a6b-df12-43cd-abc7-b0962b957ce2_isrForCrosstalkSources_998107_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:20 proc_lsst.multi:146 [INFO] found job 17093098 in provider astro proc_lsst.multi INFO: found job 17093098 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8a361ada80483d092954e7596bfa00c with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/15ce66a4-1af0-490b-a2dc-1c9aa8977ae3_isrForCrosstalkSources_998212_15.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/15ce66a4-1af0-490b-a2dc-1c9aa8977ae3_isrForCrosstalkSources_998212_15.stderr parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4bf3d2d92f04a1f893471897a223531 with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/7fe26f46-aa89-403d-95b3-e72fc287e147_isrForCrosstalkSources_998107_30.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/7fe26f46-aa89-403d-95b3-e72fc287e147_isrForCrosstalkSources_998107_30.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 188f338ee44cea11e3c2f5d4f615a6f3 with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/7f178f00-ba08-4c4d-9807-ae47b508f09a_isrForCrosstalkSources_998107_31.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/7f178f00-ba08-4c4d-9807-ae47b508f09a_isrForCrosstalkSources_998107_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:21 proc_lsst.multi:146 [INFO] found job 17093099 in provider astro proc_lsst.multi INFO: found job 17093099 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 610d238835ef50376d2637ce1b2dea02 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/3e17ec25-d0ab-469c-b804-440f5d67a633_isrForCrosstalkSources_998108_50.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/3e17ec25-d0ab-469c-b804-440f5d67a633_isrForCrosstalkSources_998108_50.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66af399809922b082c49189d72fa1266 with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/a6b66642-9faf-4597-a1ee-6ccc1f41e64c_isrForCrosstalkSources_998210_55.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/a6b66642-9faf-4597-a1ee-6ccc1f41e64c_isrForCrosstalkSources_998210_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:21 proc_lsst.multi:146 [INFO] found job 17093100 in provider astro proc_lsst.multi INFO: found job 17093100 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4817503371a3194295fc13546a97483e with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/4f330e5d-0c22-489d-b60b-0bfc7d6f2f4a_isrForCrosstalkSources_998210_22.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/4f330e5d-0c22-489d-b60b-0bfc7d6f2f4a_isrForCrosstalkSources_998210_22.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30350696ff776860b288200efa69f05e with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/5ada845f-f72a-4f12-b2c8-2ce6a6b1f387_isrForCrosstalkSources_998211_55.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/5ada845f-f72a-4f12-b2c8-2ce6a6b1f387_isrForCrosstalkSources_998211_55.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d60eed9c5653a7d5c15a1b942b431665 with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/9034778d-c4d5-418e-9512-a15c6029dfa1_isrForCrosstalkSources_998211_3.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/9034778d-c4d5-418e-9512-a15c6029dfa1_isrForCrosstalkSources_998211_3.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c6586d67a95f4da98526e1904c13fd3 with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/b119e9a7-e2b9-4c51-aab3-094480255980_isrForCrosstalkSources_998108_46.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/b119e9a7-e2b9-4c51-aab3-094480255980_isrForCrosstalkSources_998108_46.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d9319e154abf5ad17172773306d83b7 with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/d9bcb083-7c1b-4e96-98e1-70dac5f8fe42_isrForCrosstalkSources_998106_60.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/d9bcb083-7c1b-4e96-98e1-70dac5f8fe42_isrForCrosstalkSources_998106_60.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3601cbf6f9fd6ef1442b12fbb1c168c2 with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/63ae4365-280d-4ae4-a41f-7667a5d77d03_isrForCrosstalkSources_998212_5.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/63ae4365-280d-4ae4-a41f-7667a5d77d03_isrForCrosstalkSources_998212_5.stderr parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8e5f2871f5e843321e6e72e2c2c43dc with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/ce553aff-6e38-4efc-b5cb-930d2281ec2f_isrForCrosstalkSources_998106_3.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/ce553aff-6e38-4efc-b5cb-930d2281ec2f_isrForCrosstalkSources_998106_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:24 proc_lsst.multi:146 [INFO] found job 17093101 in provider astro proc_lsst.multi INFO: found job 17093101 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:24 proc_lsst.multi:146 [INFO] found job 17093102 in provider astro proc_lsst.multi INFO: found job 17093102 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc3abfe76a9d026e00b54c8c971289d2 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/d2071dec-4191-45a4-a12f-cfce8e0e3ef6_isrForCrosstalkSources_998108_45.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/d2071dec-4191-45a4-a12f-cfce8e0e3ef6_isrForCrosstalkSources_998108_45.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3946f2cf9562df671bfc1ec228b8872 with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/1e299113-c1d9-47cb-ad9f-fb7c04a94dee_isrForCrosstalkSources_998211_27.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/1e299113-c1d9-47cb-ad9f-fb7c04a94dee_isrForCrosstalkSources_998211_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:25 proc_lsst.multi:146 [INFO] found job 17093103 in provider astro proc_lsst.multi INFO: found job 17093103 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:26 proc_lsst.multi:146 [INFO] found job 17093104 in provider astro proc_lsst.multi INFO: found job 17093104 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7523579310ffb97bfb140d2a1bcdf03 with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/213b1071-e971-49d7-b4c4-f114e6b9cd91_isrForCrosstalkSources_998107_21.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/213b1071-e971-49d7-b4c4-f114e6b9cd91_isrForCrosstalkSources_998107_21.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d97ab5652539f142923157c3c536b8af with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/0bd62e64-126e-4e90-8c2c-c35e3e231fce_isrForCrosstalkSources_998106_52.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/0bd62e64-126e-4e90-8c2c-c35e3e231fce_isrForCrosstalkSources_998106_52.stderr parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b1daa90b01576835c988145affaea7b with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/422b3bcb-27db-445f-9d4f-673785ada959_isrForCrosstalkSources_998107_41.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/422b3bcb-27db-445f-9d4f-673785ada959_isrForCrosstalkSources_998107_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:27 proc_lsst.multi:146 [INFO] found job 17093105 in provider astro proc_lsst.multi INFO: found job 17093105 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30c79d88b75493fbbac52092bb1f726f with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/a02057c2-5bc7-411b-acc3-9a0d3da40359_isrForCrosstalkSources_998108_27.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/a02057c2-5bc7-411b-acc3-9a0d3da40359_isrForCrosstalkSources_998108_27.stderr parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d95c928b67040f1023349746ae20c1a with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/20f55edf-8725-4079-9e5b-120bbcf06886_isrForCrosstalkSources_998210_48.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/20f55edf-8725-4079-9e5b-120bbcf06886_isrForCrosstalkSources_998210_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:27 proc_lsst.multi:146 [INFO] found job 17093106 in provider astro proc_lsst.multi INFO: found job 17093106 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac1bfb96d3d236cd8f786d4316ac85be with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/feb4ce61-dfad-4dda-988d-80f6d72aad51_isrForCrosstalkSources_998107_59.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/feb4ce61-dfad-4dda-988d-80f6d72aad51_isrForCrosstalkSources_998107_59.stderr parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cef1354faa93441945b7e2224606f484 with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/3e4bf6ca-87ad-4248-872e-7fc54a878a1b_isrForCrosstalkSources_998108_23.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/3e4bf6ca-87ad-4248-872e-7fc54a878a1b_isrForCrosstalkSources_998108_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:28 proc_lsst.multi:146 [INFO] found job 17093107 in provider astro proc_lsst.multi INFO: found job 17093107 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:28 proc_lsst.multi:146 [INFO] found job 17093108 in provider astro proc_lsst.multi INFO: found job 17093108 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6020bc02eed38c04d7191755a3bd3bd with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/9137fde0-8da4-4957-b977-c6efce6b69a9_isrForCrosstalkSources_998107_57.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/9137fde0-8da4-4957-b977-c6efce6b69a9_isrForCrosstalkSources_998107_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:29 proc_lsst.multi:146 [INFO] found job 17093109 in provider astro proc_lsst.multi INFO: found job 17093109 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:29 proc_lsst.multi:146 [INFO] found job 17093110 in provider astro proc_lsst.multi INFO: found job 17093110 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e7494d6ff09eadca97f33e6b27dd245 with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/990a6e21-5e7b-4e73-9d1c-2211a8b628fb_isrForCrosstalkSources_998212_24.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/990a6e21-5e7b-4e73-9d1c-2211a8b628fb_isrForCrosstalkSources_998212_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:30 proc_lsst.multi:146 [INFO] found job 17093113 in provider astro proc_lsst.multi INFO: found job 17093113 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdc257a6332c4b026290b8911f9c2644 with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/5b4b7bd5-0d0f-4c58-9f7a-822da1205198_isrForCrosstalkSources_998212_59.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/5b4b7bd5-0d0f-4c58-9f7a-822da1205198_isrForCrosstalkSources_998212_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:30 proc_lsst.multi:146 [INFO] found job 17093114 in provider astro proc_lsst.multi INFO: found job 17093114 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:31 proc_lsst.multi:146 [INFO] found job 17093115 in provider astro proc_lsst.multi INFO: found job 17093115 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:31 proc_lsst.multi:146 [INFO] found job 17093116 in provider astro proc_lsst.multi INFO: found job 17093116 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8b7697ec2698d7d6b8e147aba6b9f99 with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/4cca31ce-c18e-4630-8472-f79147039b46_isrForCrosstalkSources_998106_19.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/4cca31ce-c18e-4630-8472-f79147039b46_isrForCrosstalkSources_998106_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:31 proc_lsst.multi:146 [INFO] found job 17093117 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093117 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093117,17093118,17093119,17093120,17093121' parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 245e5719fa2db2077def113703b817c6 with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/edf15cac-99ba-4162-8789-930968fd49a5_isrForCrosstalkSources_998108_8.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/edf15cac-99ba-4162-8789-930968fd49a5_isrForCrosstalkSources_998108_8.stderr parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd91e699b785a0c65d3cf1b7a6fd4a4e with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/20631956-4fdb-470c-a834-776015970c50_isrForCrosstalkSources_998107_5.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/20631956-4fdb-470c-a834-776015970c50_isrForCrosstalkSources_998107_5.stderr parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e8858cab452f88e255bf3832e7ec138 with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/a6b459d3-90ed-4c7c-8a91-2a3c50217867_isrForCrosstalkSources_998108_39.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/a6b459d3-90ed-4c7c-8a91-2a3c50217867_isrForCrosstalkSources_998108_39.stderr parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 144f7de07ce61dfa10f2ed9101085851 with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/7c64c40d-75f4-4301-a14e-fee326535477_isrForCrosstalkSources_998211_14.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/7c64c40d-75f4-4301-a14e-fee326535477_isrForCrosstalkSources_998211_14.stderr parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e2001725410051b5edba52eacff7216 with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/5bc90cee-cf4c-4253-a72b-b30d79f3a9da_isrForCrosstalkSources_998210_1.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/5bc90cee-cf4c-4253-a72b-b30d79f3a9da_isrForCrosstalkSources_998210_1.stderr parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ba03212bb0003d6f662ce9e46c02252 with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/492b5709-e289-4eb7-b6fa-16095cf61ede_isrForCrosstalkSources_998211_31.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/492b5709-e289-4eb7-b6fa-16095cf61ede_isrForCrosstalkSources_998211_31.stderr parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea26eac2dff8cf8bd495bc770310363e with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/8bf4137a-efb1-48b8-bcc1-7e50536ffec8_isrForCrosstalkSources_998212_33.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/8bf4137a-efb1-48b8-bcc1-7e50536ffec8_isrForCrosstalkSources_998212_33.stderr parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ef21e28463a7f49a50ff084a21122a2 with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/d319d748-bffa-493e-bc93-117ada503282_isrForCrosstalkSources_998107_39.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/d319d748-bffa-493e-bc93-117ada503282_isrForCrosstalkSources_998107_39.stderr parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adeec617eb96fcc89fa377c87034cbaa with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/d1126a1c-b361-4423-9efb-4775b153e34d_isrForCrosstalkSources_998108_20.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/d1126a1c-b361-4423-9efb-4775b153e34d_isrForCrosstalkSources_998108_20.stderr parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c0a70486c2d6d594e5696b1c9513965 with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/25aacc57-59f9-4c40-a28d-0fb3bb6c22a2_isrForCrosstalkSources_998210_46.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/25aacc57-59f9-4c40-a28d-0fb3bb6c22a2_isrForCrosstalkSources_998210_46.stderr parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4dfd713829d0861ce814e8fe1186515 with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/9c99b0f8-a763-419c-a1dc-3f63e7d3476d_isrForCrosstalkSources_998211_56.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/9c99b0f8-a763-419c-a1dc-3f63e7d3476d_isrForCrosstalkSources_998211_56.stderr parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ef53348d75e48588d9f48b0e510f6bd with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/9cc016b4-2bad-4678-b8ce-9ffbfda008b5_isrForCrosstalkSources_998210_17.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/9cc016b4-2bad-4678-b8ce-9ffbfda008b5_isrForCrosstalkSources_998210_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093117 R 17093118 R 17093119 R 17093120 R 17093121 R parsl.providers.slurm.slurm DEBUG: Updating job 17093117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093121 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:32 proc_lsst.multi:146 [INFO] found job 17093118 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093118 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093117,17093118,17093119,17093120,17093121' parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 377233d051343410ddced7d29ea1f07b with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/e5881988-e5fd-4096-beff-29eaf8a7dfb0_isrForCrosstalkSources_998106_21.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/e5881988-e5fd-4096-beff-29eaf8a7dfb0_isrForCrosstalkSources_998106_21.stderr parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1895f8c9de17b68b5b22ec1be3b8b9d with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/96bc77a8-b49f-4059-bfe3-2ed5e38d19ba_isrForCrosstalkSources_998212_9.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/96bc77a8-b49f-4059-bfe3-2ed5e38d19ba_isrForCrosstalkSources_998212_9.stderr parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0edaad8eedd6b8f3bc8b26b1512a1ea6 with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/2b662770-2dda-49b7-9218-b88617f9bee7_isrForCrosstalkSources_998212_3.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/2b662770-2dda-49b7-9218-b88617f9bee7_isrForCrosstalkSources_998212_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093117 R 17093118 R 17093119 R 17093120 R 17093121 R parsl.providers.slurm.slurm DEBUG: Updating job 17093117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093121 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:33 proc_lsst.multi:146 [INFO] found job 17093119 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093119 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093117,17093118,17093119,17093120,17093121' parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30e11789c5bafce697249cd861091794 with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/4f79728e-990f-4ef5-944c-6b06d57284e8_isrForCrosstalkSources_998211_61.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/4f79728e-990f-4ef5-944c-6b06d57284e8_isrForCrosstalkSources_998211_61.stderr parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 719a72672bed669dfbacf6260fc98331 with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/e4d5d3d8-9ac0-44e3-91de-6b671d071d27_isrForCrosstalkSources_998108_35.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/e4d5d3d8-9ac0-44e3-91de-6b671d071d27_isrForCrosstalkSources_998108_35.stderr parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea3b2ac2171bb0d18ec1179244a11075 with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/5355d879-559d-4de5-871e-0ee0c68c1d6f_isrForCrosstalkSources_998106_42.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/5355d879-559d-4de5-871e-0ee0c68c1d6f_isrForCrosstalkSources_998106_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093117 R 17093118 R 17093119 R 17093120 R 17093121 R parsl.providers.slurm.slurm DEBUG: Updating job 17093117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093121 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:33 proc_lsst.multi:146 [INFO] found job 17093120 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093120 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093117,17093118,17093119,17093120,17093121' parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b6aaa6923b3dc68c1ffc0d225eb1f0d with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/eb115207-35bf-444e-8809-650ac85bf1c7_isrForCrosstalkSources_998108_58.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/eb115207-35bf-444e-8809-650ac85bf1c7_isrForCrosstalkSources_998108_58.stderr parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f79b5ae2d1e57d219eda74f9e1d8ac22 with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/6b02e212-4433-4246-8a83-c79988117291_isrForCrosstalkSources_998106_47.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/6b02e212-4433-4246-8a83-c79988117291_isrForCrosstalkSources_998106_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093117 R 17093118 R 17093119 R 17093120 R 17093121 R parsl.providers.slurm.slurm DEBUG: Updating job 17093117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093121 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:34 proc_lsst.multi:146 [INFO] found job 17093121 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093121 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093117,17093118,17093119,17093120,17093121' parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dc9d7828cfc2d72aa91fd1abd09e759 with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/de44599d-a8ac-4b39-bc45-b59cc6661aff_isrForCrosstalkSources_998107_7.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/de44599d-a8ac-4b39-bc45-b59cc6661aff_isrForCrosstalkSources_998107_7.stderr parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94dcf2bd64313d5ed09932a5e8c1b7de with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/ad3613c1-79f4-4ef5-b05a-e1f45ab0b522_isrForCrosstalkSources_998107_46.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/ad3613c1-79f4-4ef5-b05a-e1f45ab0b522_isrForCrosstalkSources_998107_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093117 R 17093118 R 17093119 R 17093120 R 17093121 R parsl.providers.slurm.slurm DEBUG: Updating job 17093117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093121 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:35 proc_lsst.multi:146 [INFO] found job 17093122 in provider escience_a40 proc_lsst.multi INFO: found job 17093122 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8920fdc0b488c70bc97856d7f8a2e3ad with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/24280292-ae6f-4eff-81d7-a22e198c6511_isrForCrosstalkSources_998212_19.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/24280292-ae6f-4eff-81d7-a22e198c6511_isrForCrosstalkSources_998212_19.stderr parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a1c17547fe921fd0d7c135162aa6105 with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/2af379ee-c027-407e-9bc5-587c3918412a_isrForCrosstalkSources_998212_6.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/2af379ee-c027-407e-9bc5-587c3918412a_isrForCrosstalkSources_998212_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:35 proc_lsst.multi:146 [INFO] found job 17093123 in provider escience_a40 proc_lsst.multi INFO: found job 17093123 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:36 proc_lsst.multi:146 [INFO] found job 17093124 in provider escience_a40 proc_lsst.multi INFO: found job 17093124 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5216cc2fa1b7a437ffe1252d88b7d433 with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/00b01b71-2a84-49c7-ac59-4e8f9a99dee0_isrForCrosstalkSources_998108_49.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/00b01b71-2a84-49c7-ac59-4e8f9a99dee0_isrForCrosstalkSources_998108_49.stderr parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a56a9ead4f397a5b42d21cae6dba904f with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/06e4b068-c4f2-412c-95f8-7d3ff2f81b8f_isrForCrosstalkSources_998107_61.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/06e4b068-c4f2-412c-95f8-7d3ff2f81b8f_isrForCrosstalkSources_998107_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:36 proc_lsst.multi:146 [INFO] found job 17093125 in provider escience_a40 proc_lsst.multi INFO: found job 17093125 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d0aa7cd190077096229dc30dfdfdaae with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/2cc66ffa-12cc-4800-b60b-c82259c2651a_isrForCrosstalkSources_998107_4.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/2cc66ffa-12cc-4800-b60b-c82259c2651a_isrForCrosstalkSources_998107_4.stderr parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8e2a4e87b2b5e1ed76bfa29a89c429c with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/213d8a3e-f108-4f27-866d-eeba41df7b70_isrForCrosstalkSources_998210_15.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/213d8a3e-f108-4f27-866d-eeba41df7b70_isrForCrosstalkSources_998210_15.stderr parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 575edec2a6f6fdecc71f0066b0537759 with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/06738163-c244-4c9e-a7ec-709f962ce037_isrForCrosstalkSources_998106_34.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/06738163-c244-4c9e-a7ec-709f962ce037_isrForCrosstalkSources_998106_34.stderr parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41790b03ebed2cfe937a2fa7f7425b48 with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/24fa03c4-a4b0-4a19-9c8b-bb0729063bf0_isrForCrosstalkSources_998212_26.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/24fa03c4-a4b0-4a19-9c8b-bb0729063bf0_isrForCrosstalkSources_998212_26.stderr parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb3d31447f3bce2af8876f48c632bae4 with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/c5e20072-3d0d-425f-82db-da8c6d93d183_isrForCrosstalkSources_998107_25.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/c5e20072-3d0d-425f-82db-da8c6d93d183_isrForCrosstalkSources_998107_25.stderr parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 594084c492848c9cb59b595b6297b0ff with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/6d4f0b1d-fdba-4292-a0dc-7e9c2103f805_isrForCrosstalkSources_998212_27.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/6d4f0b1d-fdba-4292-a0dc-7e9c2103f805_isrForCrosstalkSources_998212_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:36 proc_lsst.multi:146 [INFO] found job 17093126 in provider escience_a40 proc_lsst.multi INFO: found job 17093126 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0704c7e00234bf039932afd9ee8c316b with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/403f5852-4d9c-402b-a024-60a4375dabae_isrForCrosstalkSources_998107_51.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/403f5852-4d9c-402b-a024-60a4375dabae_isrForCrosstalkSources_998107_51.stderr parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9270438be8272ed2847cf8eddad55e1 with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/1bea362c-d4be-4735-b0eb-1ae26d040b38_isrForCrosstalkSources_998211_39.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/1bea362c-d4be-4735-b0eb-1ae26d040b38_isrForCrosstalkSources_998211_39.stderr parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 517984e43719ae2bde89a439f874d87b with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/f0c786b9-e7a0-4940-bf74-aad384c05a83_isrForCrosstalkSources_998211_19.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/f0c786b9-e7a0-4940-bf74-aad384c05a83_isrForCrosstalkSources_998211_19.stderr parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd0297266c93f64f19ebbcd8f09bdcf2 with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/a36475b7-b60a-4540-af70-88012920a296_isrForCrosstalkSources_998211_15.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/a36475b7-b60a-4540-af70-88012920a296_isrForCrosstalkSources_998211_15.stderr parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c59732b676a653f7fc00326658b39e4 with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/2a2c0d71-4635-406c-989f-a5e0c928994c_isrForCrosstalkSources_998107_40.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/2a2c0d71-4635-406c-989f-a5e0c928994c_isrForCrosstalkSources_998107_40.stderr parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6d1e417b8ddf4323d308e6d0099ce7c with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/79efce15-b5d5-4237-8ed5-2addb197daf6_isrForCrosstalkSources_998106_44.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/79efce15-b5d5-4237-8ed5-2addb197daf6_isrForCrosstalkSources_998106_44.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa3bfc5f040f0af4dc79c722d3fa68e2 with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/5a0aa601-a5c8-4f51-8830-943a6eac3a44_isrForCrosstalkSources_998106_14.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/5a0aa601-a5c8-4f51-8830-943a6eac3a44_isrForCrosstalkSources_998106_14.stderr parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dccb0d47c7b305ad3191e26221263701 with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/278cdfb2-7dd5-494d-809b-6feed3a45aca_isrForCrosstalkSources_998212_53.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/278cdfb2-7dd5-494d-809b-6feed3a45aca_isrForCrosstalkSources_998212_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:37 proc_lsst.multi:146 [INFO] found job 17093127 in provider escience_a40 proc_lsst.multi INFO: found job 17093127 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b82a67eae199142b9bda4783afc7af3 with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/a6f14c72-be9c-4902-8087-9134bfe68482_isrForCrosstalkSources_998106_43.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/a6f14c72-be9c-4902-8087-9134bfe68482_isrForCrosstalkSources_998106_43.stderr parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cc79fdbdcc8bf5a17419d9dad80a90c with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/76a355a1-b4f8-4bc5-8091-cd7bf4326688_isrForCrosstalkSources_998106_36.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/76a355a1-b4f8-4bc5-8091-cd7bf4326688_isrForCrosstalkSources_998106_36.stderr parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6196e311ea6bb0b62754cefb655a8541 with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/4f34e4f8-d881-4628-ba9f-cae7892d0bae_isrForCrosstalkSources_998107_9.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/4f34e4f8-d881-4628-ba9f-cae7892d0bae_isrForCrosstalkSources_998107_9.stderr parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de1b62772cb6deb78a2881676804e8b8 with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/9ef6f49a-b34b-4b70-9774-e79b4d68c914_isrForCrosstalkSources_998212_36.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/9ef6f49a-b34b-4b70-9774-e79b4d68c914_isrForCrosstalkSources_998212_36.stderr parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5fc3327a1ee7e07c8908ef8f4b3789e with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/7cf048ed-549d-4a3b-9092-826625ed733c_isrForCrosstalkSources_998106_38.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/7cf048ed-549d-4a3b-9092-826625ed733c_isrForCrosstalkSources_998106_38.stderr parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1ccb948a6fdb1a4ea40902b7ff89a06 with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/909dcc40-5375-42c6-b21a-36bdcd6f1ef0_isrForCrosstalkSources_998106_50.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/909dcc40-5375-42c6-b21a-36bdcd6f1ef0_isrForCrosstalkSources_998106_50.stderr parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 145c8cfe13f217a3c0725aa765e95393 with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/e2447ac6-eebb-4658-ae58-abb7df9ca1cf_isrForCrosstalkSources_998108_61.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/e2447ac6-eebb-4658-ae58-abb7df9ca1cf_isrForCrosstalkSources_998108_61.stderr parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ace8be516d54cf75507aa611c7241a07 with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/6f7efc4b-7738-4c41-90fa-a8cbfb9ce32a_isrForCrosstalkSources_998210_23.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/6f7efc4b-7738-4c41-90fa-a8cbfb9ce32a_isrForCrosstalkSources_998210_23.stderr parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a86de2052381e1b54d3d8d43732f2a13 with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/62e08bb1-ba2d-45a5-86af-ca5012bdf78f_isrForCrosstalkSources_998108_44.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/62e08bb1-ba2d-45a5-86af-ca5012bdf78f_isrForCrosstalkSources_998108_44.stderr parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c622acd63c95ea1b69a71259ca3af28 with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/e2200ead-d0ab-422c-bd64-af6b09305341_isrForCrosstalkSources_998108_53.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/e2200ead-d0ab-422c-bd64-af6b09305341_isrForCrosstalkSources_998108_53.stderr parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 291289a09898b2323b2e88c4272d597c with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/16799f2f-914a-466b-9db0-5ffe844b798d_isrForCrosstalkSources_998106_49.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/16799f2f-914a-466b-9db0-5ffe844b798d_isrForCrosstalkSources_998106_49.stderr parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb09926ab98bd3eea98f013cd28adf8b with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/26a4e535-99a5-4433-bfb4-e01fe9a67b29_isrForCrosstalkSources_998211_58.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/26a4e535-99a5-4433-bfb4-e01fe9a67b29_isrForCrosstalkSources_998211_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:37 proc_lsst.multi:146 [INFO] found job 17093128 in provider escience_a40 proc_lsst.multi INFO: found job 17093128 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69fee44679029a98fd395f7bc57e2d2f with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/9f519d04-1eea-489b-beab-e05730b6685a_isrForCrosstalkSources_998212_50.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/9f519d04-1eea-489b-beab-e05730b6685a_isrForCrosstalkSources_998212_50.stderr parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abec207e69eab9f30b2274e1cfecef9d with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/acf263cb-18dc-4b19-b01d-438f5617d45c_isrForCrosstalkSources_998107_47.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/acf263cb-18dc-4b19-b01d-438f5617d45c_isrForCrosstalkSources_998107_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:38 proc_lsst.multi:146 [INFO] found job 17093129 in provider escience_a40 proc_lsst.multi INFO: found job 17093129 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:39 proc_lsst.multi:146 [INFO] found job 17093130 in provider escience_a40 proc_lsst.multi INFO: found job 17093130 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:39 proc_lsst.multi:146 [INFO] found job 17093132 in provider escience_a40 proc_lsst.multi INFO: found job 17093132 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:40 proc_lsst.multi:146 [INFO] found job 17093133 in provider escience_a40 proc_lsst.multi INFO: found job 17093133 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cf1ce25a474689932451f973db5b8bf with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/1637a91c-ea80-43dd-bb8a-4bab0abcce74_isrForCrosstalkSources_998211_34.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/1637a91c-ea80-43dd-bb8a-4bab0abcce74_isrForCrosstalkSources_998211_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:40 proc_lsst.multi:146 [INFO] found job 17093135 in provider escience_a40 proc_lsst.multi INFO: found job 17093135 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:41 proc_lsst.multi:146 [INFO] found job 17093136 in provider escience_a40 proc_lsst.multi INFO: found job 17093136 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39db09ba7fc456375abcc685bf259aa6 with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/500d6c2d-b30c-48fd-bf9e-dd5f5a3c9571_isrForCrosstalkSources_998210_14.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/500d6c2d-b30c-48fd-bf9e-dd5f5a3c9571_isrForCrosstalkSources_998210_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:41 proc_lsst.multi:146 [INFO] found job 17093138 in provider escience_a40 proc_lsst.multi INFO: found job 17093138 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:41 proc_lsst.multi:146 [INFO] found job 17093140 in provider escience_a40 proc_lsst.multi INFO: found job 17093140 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:41 proc_lsst.multi:146 [INFO] found job 17093142 in provider escience_a40 proc_lsst.multi INFO: found job 17093142 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:42 proc_lsst.multi:146 [INFO] found job 17093143 in provider escience_a40 proc_lsst.multi INFO: found job 17093143 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:42 proc_lsst.multi:146 [INFO] found job 17093145 in provider escience_a40 proc_lsst.multi INFO: found job 17093145 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:42 proc_lsst.multi:146 [INFO] found job 17093146 in provider escience_a40 proc_lsst.multi INFO: found job 17093146 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:43 proc_lsst.multi:146 [INFO] found job 17093147 in provider escience_a40 proc_lsst.multi INFO: found job 17093147 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:43 proc_lsst.multi:146 [INFO] found job 17093149 in provider checkpoint proc_lsst.multi INFO: found job 17093149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 274326c1ad07c8abc14995c5b217b0bf with result from task 262 parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/162bf8db-74a7-4a17-9e5f-750254ec44fa_isrForCrosstalkSources_998106_23.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/162bf8db-74a7-4a17-9e5f-750254ec44fa_isrForCrosstalkSources_998106_23.stderr parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71d9b944fd1ec45261b5f535b4da2712 with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/297620a5-e7c1-44e2-b3fb-ef01eba9eea7_isrForCrosstalkSources_998106_45.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/297620a5-e7c1-44e2-b3fb-ef01eba9eea7_isrForCrosstalkSources_998106_45.stderr parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 363cdbc3596adb930f83e5cf030e6436 with result from task 266 parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/322552ad-71f6-4cfb-83c4-ebae04cf35e4_isrForCrosstalkSources_998106_24.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/322552ad-71f6-4cfb-83c4-ebae04cf35e4_isrForCrosstalkSources_998106_24.stderr parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 546da228fe4b74c533186a2bedab2fa6 with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/b390a8b5-1cd5-457b-8a49-432c08e811b6_isrForCrosstalkSources_998107_27.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/b390a8b5-1cd5-457b-8a49-432c08e811b6_isrForCrosstalkSources_998107_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:43 proc_lsst.multi:146 [INFO] found job 17093150 in provider checkpoint proc_lsst.multi INFO: found job 17093150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3b05b7e3e6a9cb7943d2ac0fe6a4066 with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/0faa5d51-d329-4411-b22c-a1d45ad2055c_isrForCrosstalkSources_998106_9.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/0faa5d51-d329-4411-b22c-a1d45ad2055c_isrForCrosstalkSources_998106_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:44 proc_lsst.multi:146 [INFO] found job 17093151 in provider checkpoint proc_lsst.multi INFO: found job 17093151 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 564c3cccc615242958d80d6312c72359 with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/21c29fd7-ea1f-4b73-872b-07edf107e9f3_isrForCrosstalkSources_998107_45.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/21c29fd7-ea1f-4b73-872b-07edf107e9f3_isrForCrosstalkSources_998107_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:44 proc_lsst.multi:146 [INFO] found job 17093153 in provider checkpoint proc_lsst.multi INFO: found job 17093153 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 746e5eeefb37992f5eb23cb7fc5c2596 with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/ebb28cfa-7e9c-4a00-9a73-ed9844c9955f_isrForCrosstalkSources_998211_8.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/ebb28cfa-7e9c-4a00-9a73-ed9844c9955f_isrForCrosstalkSources_998211_8.stderr parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3311df3deff562241362294b78c484c with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/8ad5a76f-6f21-47cc-bdab-1fb4e7d1a6f8_isrForCrosstalkSources_998211_51.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/8ad5a76f-6f21-47cc-bdab-1fb4e7d1a6f8_isrForCrosstalkSources_998211_51.stderr parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 183349d73451a4c4eb800ab8d4725d8e with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/82a4c72a-ee6b-4401-9b11-f20c7738fda7_isrForCrosstalkSources_998211_43.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/82a4c72a-ee6b-4401-9b11-f20c7738fda7_isrForCrosstalkSources_998211_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:44 proc_lsst.multi:146 [INFO] found job 17093154 in provider checkpoint proc_lsst.multi INFO: found job 17093154 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 421a31c84a7737e8547a45bf9c9424ad with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/0b2b9cc6-77e9-4d96-a637-de4eb7c6a139_isrForCrosstalkSources_998211_53.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/0b2b9cc6-77e9-4d96-a637-de4eb7c6a139_isrForCrosstalkSources_998211_53.stderr parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3755fe179546b8b3e8ca84084af32eea with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/45272384-07c5-4430-a32c-d5099ea60c52_isrForCrosstalkSources_998211_11.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/45272384-07c5-4430-a32c-d5099ea60c52_isrForCrosstalkSources_998211_11.stderr parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b394fec2e7f45ca31b072e7791b35ca3 with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/0761ff90-1dad-487f-a850-ae2e01d2a203_isrForCrosstalkSources_998210_35.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/0761ff90-1dad-487f-a850-ae2e01d2a203_isrForCrosstalkSources_998210_35.stderr parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55afd4fde77bc28babcfb7325d53dd11 with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/df8bf773-b056-4ad5-a248-43c10d0b0e9b_isrForCrosstalkSources_998108_41.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/df8bf773-b056-4ad5-a248-43c10d0b0e9b_isrForCrosstalkSources_998108_41.stderr parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9586df0c7ce533610c722c29ba0e8355 with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/304ad50c-49f7-4573-a0f0-f6d3cab741cb_isrForCrosstalkSources_998108_36.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/304ad50c-49f7-4573-a0f0-f6d3cab741cb_isrForCrosstalkSources_998108_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:45 proc_lsst.multi:146 [INFO] found job 17093155 in provider checkpoint proc_lsst.multi INFO: found job 17093155 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4595ef186a48114fa906978091bfe644 with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/d82624fa-fe3a-4ef6-8ee2-32a3e14585b5_isrForCrosstalkSources_998211_49.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/d82624fa-fe3a-4ef6-8ee2-32a3e14585b5_isrForCrosstalkSources_998211_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:45 proc_lsst.multi:146 [INFO] found job 17093157 in provider checkpoint proc_lsst.multi INFO: found job 17093157 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89816acc6f766db3e5fa3fa41cb6c035 with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/17cdcac1-a0f0-41dc-81b4-aaf1ae4546bb_isrForCrosstalkSources_998212_42.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/17cdcac1-a0f0-41dc-81b4-aaf1ae4546bb_isrForCrosstalkSources_998212_42.stderr parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c15a93342bf120a251b4fb93509746c9 with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/012f14bd-61e6-4d46-976e-f8ecbbc51b74_isrForCrosstalkSources_998108_28.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/012f14bd-61e6-4d46-976e-f8ecbbc51b74_isrForCrosstalkSources_998108_28.stderr parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d699585e44393aab133886f5df962d61 with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/63b68206-41b2-4396-a788-92ea1fe72118_isrForCrosstalkSources_998210_54.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/63b68206-41b2-4396-a788-92ea1fe72118_isrForCrosstalkSources_998210_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:45 proc_lsst.multi:146 [INFO] found job 17093158 in provider checkpoint proc_lsst.multi INFO: found job 17093158 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a6b998c641bb82a37398a3143b81eab with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/ab0b52a3-6850-49de-9952-5c75977f3ebf_isrForCrosstalkSources_998210_52.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/ab0b52a3-6850-49de-9952-5c75977f3ebf_isrForCrosstalkSources_998210_52.stderr parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be720451a21e621c5102827fb951b349 with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/a3cb58f3-47a7-4072-84f4-34ffd616d227_isrForCrosstalkSources_998210_58.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/a3cb58f3-47a7-4072-84f4-34ffd616d227_isrForCrosstalkSources_998210_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:45 proc_lsst.multi:146 [INFO] found job 17093159 in provider checkpoint proc_lsst.multi INFO: found job 17093159 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0e1e93e65e4ae655a9118275db799ba with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/1501d4e1-8a40-461e-a521-a4cc1b994522_isrForCrosstalkSources_998212_10.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/1501d4e1-8a40-461e-a521-a4cc1b994522_isrForCrosstalkSources_998212_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:46 proc_lsst.multi:146 [INFO] found job 17093160 in provider checkpoint proc_lsst.multi INFO: found job 17093160 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e3f69f4572359745bdcbe4bd8b5f892 with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/a7d8996b-3863-47b5-97d1-2764cfe42064_isrForCrosstalkSources_998212_34.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/a7d8996b-3863-47b5-97d1-2764cfe42064_isrForCrosstalkSources_998212_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:46 proc_lsst.multi:146 [INFO] found job 17093161 in provider checkpoint proc_lsst.multi INFO: found job 17093161 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36141c2a6ed5c560522f211394cb86a7 with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/bc65387a-3ef0-4172-b282-4b28221eba3d_isrForCrosstalkSources_998108_55.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/bc65387a-3ef0-4172-b282-4b28221eba3d_isrForCrosstalkSources_998108_55.stderr parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0bd3025765139e6b47083e44ca9b2de with result from task 288 parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/d20cc9d1-5108-462c-9564-e0c3327e44a3_isrForCrosstalkSources_998212_41.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/d20cc9d1-5108-462c-9564-e0c3327e44a3_isrForCrosstalkSources_998212_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:46 proc_lsst.multi:146 [INFO] found job 17093162 in provider checkpoint proc_lsst.multi INFO: found job 17093162 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c935a420f2381fafb25ee8c8e40f432 with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/93061d01-4afd-4e7c-8929-7e5ca6f9c9fe_isrForCrosstalkSources_998212_55.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/93061d01-4afd-4e7c-8929-7e5ca6f9c9fe_isrForCrosstalkSources_998212_55.stderr parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b854e00178c0ff7837550089e6ec5c21 with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/842dfabd-60b1-4e89-9d56-868b60d489d5_isrForCrosstalkSources_998212_7.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/842dfabd-60b1-4e89-9d56-868b60d489d5_isrForCrosstalkSources_998212_7.stderr parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a290513a2d2c21b2016b4435f3fa683a with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/efe8389e-abe5-454f-8dbd-969adf99b352_isrForCrosstalkSources_998108_62.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/efe8389e-abe5-454f-8dbd-969adf99b352_isrForCrosstalkSources_998108_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:46 proc_lsst.multi:146 [INFO] found job 17093163 in provider checkpoint proc_lsst.multi INFO: found job 17093163 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c60b5610b4f885d0c353ab27c5a3133f with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/4cdecb8d-7f37-4b8b-9493-177541b981cd_isrForCrosstalkSources_998106_15.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/4cdecb8d-7f37-4b8b-9493-177541b981cd_isrForCrosstalkSources_998106_15.stderr parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c6ca9625b598bf42dd4960effe10be9 with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/c5a4870e-43ae-4d48-aae7-ee572e416317_isrForCrosstalkSources_998106_57.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/c5a4870e-43ae-4d48-aae7-ee572e416317_isrForCrosstalkSources_998106_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:47 proc_lsst.multi:146 [INFO] found job 17093164 in provider checkpoint proc_lsst.multi INFO: found job 17093164 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:47 proc_lsst.multi:146 [INFO] found job 17093165 in provider checkpoint proc_lsst.multi INFO: found job 17093165 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82069a7126edfc7b323e74b180e0a442 with result from task 265 parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/4af724b7-2872-4c3a-bb4a-08bfa4a693e4_isrForCrosstalkSources_998212_54.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/4af724b7-2872-4c3a-bb4a-08bfa4a693e4_isrForCrosstalkSources_998212_54.stderr parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79957d607befd1363edc6d7cf824f13b with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/bfcd9c85-cfd6-459b-9961-f1722df5f117_isrForCrosstalkSources_998212_20.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/bfcd9c85-cfd6-459b-9961-f1722df5f117_isrForCrosstalkSources_998212_20.stderr parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1bad638b957592d01556848fa71d811 with result from task 268 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/ebd46c56-d350-4f28-a644-247d6c6e245d_isrForCrosstalkSources_998106_62.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/ebd46c56-d350-4f28-a644-247d6c6e245d_isrForCrosstalkSources_998106_62.stderr parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da95e01234c6da9416d2b3ab82a09262 with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/6a394dc5-9472-46a0-afbf-168e62da50e8_isrForCrosstalkSources_998106_53.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/6a394dc5-9472-46a0-afbf-168e62da50e8_isrForCrosstalkSources_998106_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:47 proc_lsst.multi:146 [INFO] found job 17093166 in provider checkpoint proc_lsst.multi INFO: found job 17093166 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b971ab94597f3191e077737ae930660a with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/a7cfbdce-2569-410d-902e-47452775fd63_isrForCrosstalkSources_998107_36.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/a7cfbdce-2569-410d-902e-47452775fd63_isrForCrosstalkSources_998107_36.stderr parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be8fa8089709ecb010f58965d93d0ec2 with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/d06c7bbe-5f50-49fd-bf9f-170109447e02_isrForCrosstalkSources_998210_2.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/d06c7bbe-5f50-49fd-bf9f-170109447e02_isrForCrosstalkSources_998210_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:47 proc_lsst.multi:146 [INFO] found job 17093167 in provider checkpoint proc_lsst.multi INFO: found job 17093167 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:47 proc_lsst.multi:146 [INFO] found job 17093168 in provider checkpoint proc_lsst.multi INFO: found job 17093168 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3175fa531561eafc1c13763f1561dde7 with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/633ec7d0-b2a3-4738-9214-b9feaa94d4be_isrForCrosstalkSources_998210_7.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/633ec7d0-b2a3-4738-9214-b9feaa94d4be_isrForCrosstalkSources_998210_7.stderr parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66c330fc02db77267aa8dd9bca580847 with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/3196228e-e312-441d-aa56-04a9f6c68d30_isrForCrosstalkSources_998108_18.stdout parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/3196228e-e312-441d-aa56-04a9f6c68d30_isrForCrosstalkSources_998108_18.stderr parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:48 proc_lsst.multi:146 [INFO] found job 17093169 in provider checkpoint proc_lsst.multi INFO: found job 17093169 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:48 proc_lsst.multi:146 [INFO] found job 17093170 in provider checkpoint proc_lsst.multi INFO: found job 17093170 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9afdc1210b8412161f16ff1c19230cb7 with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/999e6821-67e8-4c6f-bbbc-89d5adbe6b39_isrForCrosstalkSources_998107_2.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/999e6821-67e8-4c6f-bbbc-89d5adbe6b39_isrForCrosstalkSources_998107_2.stderr parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c41f6b43aefd09b855e2566b2542a62 with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/3c5e8f7e-0e29-4d06-8beb-026269fe6e1b_isrForCrosstalkSources_998210_16.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/3c5e8f7e-0e29-4d06-8beb-026269fe6e1b_isrForCrosstalkSources_998210_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:48 proc_lsst.multi:146 [INFO] found job 17093171 in provider checkpoint proc_lsst.multi INFO: found job 17093171 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.dataflow.memoization DEBUG: Storing app cache entry e52d96a6076258473fa3f961ea72b41f with result from task 322 parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/608856b2-cac9-4c09-a7cc-ea6eb2728e8f_isrForCrosstalkSources_998107_26.stdout parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/608856b2-cac9-4c09-a7cc-ea6eb2728e8f_isrForCrosstalkSources_998107_26.stderr parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:48 proc_lsst.multi:146 [INFO] found job 17093172 in provider checkpoint proc_lsst.multi INFO: found job 17093172 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c0e87fc636db19b6e4278d5a92b9a30 with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/2ad18b02-2107-4c23-a37e-48f9d3adf516_isrForCrosstalkSources_998106_32.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/2ad18b02-2107-4c23-a37e-48f9d3adf516_isrForCrosstalkSources_998106_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:49 proc_lsst.multi:146 [INFO] found job 17093173 in provider checkpoint proc_lsst.multi INFO: found job 17093173 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5803f8bfff1bd02cdf7fa01520403b10 with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/6747991f-5486-4de1-8a7e-4bf776291fc2_isrForCrosstalkSources_998211_2.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/6747991f-5486-4de1-8a7e-4bf776291fc2_isrForCrosstalkSources_998211_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:49 proc_lsst.multi:146 [INFO] found job 17093174 in provider checkpoint proc_lsst.multi INFO: found job 17093174 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e48f998c02b2c32326c613a728c0811d with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/ccf6d565-cc17-45ca-89d8-0efddadf3004_isrForCrosstalkSources_998211_7.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/ccf6d565-cc17-45ca-89d8-0efddadf3004_isrForCrosstalkSources_998211_7.stderr parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6bd5acdb21c075adf1a5373f732479e with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/ac1495fc-8a50-4e61-9204-87010421cb4f_isrForCrosstalkSources_998107_16.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/ac1495fc-8a50-4e61-9204-87010421cb4f_isrForCrosstalkSources_998107_16.stderr parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cc66ee8f31130d16f9ae4740039b6c0 with result from task 323 parsl.dataflow.dflow DEBUG: Task 323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/29bdd4a9-1a7b-492f-87fa-2e9f18e6cd75_isrForCrosstalkSources_998212_45.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/29bdd4a9-1a7b-492f-87fa-2e9f18e6cd75_isrForCrosstalkSources_998212_45.stderr parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbee717e28bbe6f05ae25df30e050f3b with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/cbc042b2-10b4-4562-bc40-003ceaa1b343_isrForCrosstalkSources_998106_26.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/cbc042b2-10b4-4562-bc40-003ceaa1b343_isrForCrosstalkSources_998106_26.stderr parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a8ce5be83637bacc65855adab52eb68 with result from task 308 parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/521b04b5-c249-4e72-888d-5388fb1302f0_isrForCrosstalkSources_998107_13.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/521b04b5-c249-4e72-888d-5388fb1302f0_isrForCrosstalkSources_998107_13.stderr parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14b4b213f773b40b325d0dbce77853aa with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/9fb06955-ed8d-419a-ac52-e1152ac98988_isrForCrosstalkSources_998106_20.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/9fb06955-ed8d-419a-ac52-e1152ac98988_isrForCrosstalkSources_998106_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:49 proc_lsst.multi:146 [INFO] found job 17093175 in provider checkpoint proc_lsst.multi INFO: found job 17093175 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 845427ac8b7e7835b33b79ebae4f977f with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/dcd8f7a8-25c1-4881-a301-6f1b92410a46_isrForCrosstalkSources_998107_6.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/dcd8f7a8-25c1-4881-a301-6f1b92410a46_isrForCrosstalkSources_998107_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:49 proc_lsst.multi:146 [INFO] found job 17093176 in provider checkpoint proc_lsst.multi INFO: found job 17093176 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33b3b6611450ee961bc06c2779be672f with result from task 307 parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/1d8347c1-0c1f-43f6-bf66-169926410ed0_isrForCrosstalkSources_998212_30.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/1d8347c1-0c1f-43f6-bf66-169926410ed0_isrForCrosstalkSources_998212_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:49 proc_lsst.multi:146 [INFO] found job 17093177 in provider checkpoint proc_lsst.multi INFO: found job 17093177 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 013b21366468ae61f6d576a987d776e7 with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/3e4db861-1aab-46a9-a69a-fc92273b4982_isrForCrosstalkSources_998106_61.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/3e4db861-1aab-46a9-a69a-fc92273b4982_isrForCrosstalkSources_998106_61.stderr parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a69475ba70b5011f7187216b430c6cad with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/89b45a64-8c3e-45b4-8757-6fc37866b8c3_isrForCrosstalkSources_998212_62.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/89b45a64-8c3e-45b4-8757-6fc37866b8c3_isrForCrosstalkSources_998212_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:50 proc_lsst.multi:146 [INFO] found job 17093178 in provider checkpoint proc_lsst.multi INFO: found job 17093178 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:50 proc_lsst.multi:146 [INFO] found job 17093179 in provider checkpoint proc_lsst.multi INFO: found job 17093179 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:50 proc_lsst.multi:146 [INFO] found job 17093180 in provider checkpoint proc_lsst.multi INFO: found job 17093180 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4498724de7df40fcbab6330d77b94e3a with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/e58e99ff-ff65-4687-ac16-4824b02c8b33_isrForCrosstalkSources_998210_24.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/e58e99ff-ff65-4687-ac16-4824b02c8b33_isrForCrosstalkSources_998210_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:50 proc_lsst.multi:146 [INFO] found job 17093181 in provider checkpoint proc_lsst.multi INFO: found job 17093181 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:50 proc_lsst.multi:146 [INFO] found job 17093182 in provider checkpoint proc_lsst.multi INFO: found job 17093182 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:51 proc_lsst.multi:146 [INFO] found job 17093183 in provider checkpoint proc_lsst.multi INFO: found job 17093183 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:51 proc_lsst.multi:146 [INFO] found job 17093184 in provider checkpoint proc_lsst.multi INFO: found job 17093184 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:51 proc_lsst.multi:146 [INFO] found job 17093185 in provider checkpoint proc_lsst.multi INFO: found job 17093185 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:51 proc_lsst.multi:146 [INFO] found job 17093186 in provider checkpoint proc_lsst.multi INFO: found job 17093186 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:52 proc_lsst.multi:146 [INFO] found job 17093187 in provider checkpoint proc_lsst.multi INFO: found job 17093187 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 66 parsl.dataflow.strategy DEBUG: Executor multi has 66 active tasks, 66/27 running/pending blocks, and 137 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22773982342992 2024-03-15 22:48:52 proc_lsst.multi:146 [INFO] found job 36385 in provider local proc_lsst.multi INFO: found job 36385 in provider local 2024-03-15 22:48:52 proc_lsst.multi:146 [INFO] found job 17093085 in provider astro proc_lsst.multi INFO: found job 17093085 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:52 proc_lsst.multi:146 [INFO] found job 17093086 in provider astro proc_lsst.multi INFO: found job 17093086 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:52 proc_lsst.multi:146 [INFO] found job 17093087 in provider astro proc_lsst.multi INFO: found job 17093087 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:53 proc_lsst.multi:146 [INFO] found job 17093088 in provider astro proc_lsst.multi INFO: found job 17093088 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:53 proc_lsst.multi:146 [INFO] found job 17093089 in provider astro proc_lsst.multi INFO: found job 17093089 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:53 proc_lsst.multi:146 [INFO] found job 17093090 in provider astro proc_lsst.multi INFO: found job 17093090 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f97d222f331a9daaca76e92e242eb43 with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/f8c3ab26-add9-4189-8e67-f5055757a1ce_isrForCrosstalkSources_998210_39.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/f8c3ab26-add9-4189-8e67-f5055757a1ce_isrForCrosstalkSources_998210_39.stderr parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43ccdec4feac7e800c74b537dc9f4954 with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/f644d4da-d287-48c5-9780-d2b770cb9015_isrForCrosstalkSources_998210_27.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/f644d4da-d287-48c5-9780-d2b770cb9015_isrForCrosstalkSources_998210_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:53 proc_lsst.multi:146 [INFO] found job 17093091 in provider astro proc_lsst.multi INFO: found job 17093091 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5162f6ba37785000559017df2dcb9691 with result from task 296 parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/979b44f1-56a9-4ab5-b0fc-da90c089f8a4_isrForCrosstalkSources_998106_13.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/979b44f1-56a9-4ab5-b0fc-da90c089f8a4_isrForCrosstalkSources_998106_13.stderr parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b6fd9fe84a9cec896b7c95d1b6f6384 with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/523c94db-3532-47f4-b4e3-f241fb6ebef8_isrForCrosstalkSources_998210_38.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/523c94db-3532-47f4-b4e3-f241fb6ebef8_isrForCrosstalkSources_998210_38.stderr parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 281dacab8f3fad924b4569a115ecc276 with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/e96743cd-e6db-4605-8cf9-95516325461b_isrForCrosstalkSources_998210_49.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/e96743cd-e6db-4605-8cf9-95516325461b_isrForCrosstalkSources_998210_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:53 proc_lsst.multi:146 [INFO] found job 17093092 in provider astro proc_lsst.multi INFO: found job 17093092 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:54 proc_lsst.multi:146 [INFO] found job 17093093 in provider astro proc_lsst.multi INFO: found job 17093093 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:54 proc_lsst.multi:146 [INFO] found job 17093094 in provider astro proc_lsst.multi INFO: found job 17093094 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:54 proc_lsst.multi:146 [INFO] found job 17093095 in provider astro proc_lsst.multi INFO: found job 17093095 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:54 proc_lsst.multi:146 [INFO] found job 17093096 in provider astro proc_lsst.multi INFO: found job 17093096 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcbebf280950518bd19eaf562ed03554 with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/db6c609c-8098-40ce-b917-5df55238f842_isrForCrosstalkSources_998212_56.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/db6c609c-8098-40ce-b917-5df55238f842_isrForCrosstalkSources_998212_56.stderr parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a079fa0c323cfdb60b2079f9eb75b913 with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/62c737e3-f590-429c-bbca-074257cf2176_isrForCrosstalkSources_998212_38.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/62c737e3-f590-429c-bbca-074257cf2176_isrForCrosstalkSources_998212_38.stderr parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9f931b95ef0f3b41cc619bd3c6c3e0f with result from task 300 parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:54 proc_lsst.multi:146 [INFO] found job 17093097 in provider astro proc_lsst.multi INFO: found job 17093097 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/f56d6149-b396-4cac-8fef-1ca55daa58e2_isrForCrosstalkSources_998212_12.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/f56d6149-b396-4cac-8fef-1ca55daa58e2_isrForCrosstalkSources_998212_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:55 proc_lsst.multi:146 [INFO] found job 17093098 in provider astro proc_lsst.multi INFO: found job 17093098 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:55 proc_lsst.multi:146 [INFO] found job 17093099 in provider astro proc_lsst.multi INFO: found job 17093099 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e1b9c66cba3ff32c64cb1e6dfa4b9de with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/d912e4b2-4bf4-433c-b51e-1458aaf8c489_isrForCrosstalkSources_998211_38.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/d912e4b2-4bf4-433c-b51e-1458aaf8c489_isrForCrosstalkSources_998211_38.stderr parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae458580b2f370df1dd114594a033156 with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/2d38db2e-3ce8-4541-8b55-5deb711ed48e_isrForCrosstalkSources_998211_52.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/2d38db2e-3ce8-4541-8b55-5deb711ed48e_isrForCrosstalkSources_998211_52.stderr parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0884b543bcf068dfd151fac65da11f6d with result from task 320 parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/81da7d2b-c892-4da0-886b-3510d099e1cc_isrForCrosstalkSources_998107_28.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/81da7d2b-c892-4da0-886b-3510d099e1cc_isrForCrosstalkSources_998107_28.stderr parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ca4c805cbbacd8629f8f9deda932c3e with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/337a5853-d946-431d-8359-80ab18118dfb_isrForCrosstalkSources_998107_35.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/337a5853-d946-431d-8359-80ab18118dfb_isrForCrosstalkSources_998107_35.stderr parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ceeeeb5f27dfbbdc4749d766a9f3a96 with result from task 289 parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/cf296b05-9b6e-4dbd-928d-20f1e139f820_isrForCrosstalkSources_998107_60.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/cf296b05-9b6e-4dbd-928d-20f1e139f820_isrForCrosstalkSources_998107_60.stderr parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e4e63ade00d77a7713b3281850af0a4 with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/8e4e33f6-2c1e-46e7-a7e3-311ebc4ba78e_isrForCrosstalkSources_998107_62.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/8e4e33f6-2c1e-46e7-a7e3-311ebc4ba78e_isrForCrosstalkSources_998107_62.stderr parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02adb19e87e028cabfe94635e5dd436d with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/435ca4ad-528c-438e-b1f4-f0c915b8c25e_isrForCrosstalkSources_998106_31.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/435ca4ad-528c-438e-b1f4-f0c915b8c25e_isrForCrosstalkSources_998106_31.stderr parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8e903aa1a991edd0cff2118bb58b47b with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/e39de0c9-7b9a-4ee2-8cca-67ae3d533706_isrForCrosstalkSources_998107_52.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/e39de0c9-7b9a-4ee2-8cca-67ae3d533706_isrForCrosstalkSources_998107_52.stderr parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db2177afbdc506f56b0ed7379c8aa947 with result from task 310 parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/0ef607d7-b319-46d4-b84c-f8d4957f37f4_isrForCrosstalkSources_998106_41.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/0ef607d7-b319-46d4-b84c-f8d4957f37f4_isrForCrosstalkSources_998106_41.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f167f8cbe1f5912f3e6ecc5ef68bc40c with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/b639afbd-0505-4646-8156-b2900a96cc19_isrForCrosstalkSources_998106_48.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/b639afbd-0505-4646-8156-b2900a96cc19_isrForCrosstalkSources_998106_48.stderr parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b384b2082194b4dc47a7e1dec9dfbae with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/0f27bc96-0c0d-4011-a836-3e4e6d5db5c3_isrForCrosstalkSources_998108_43.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/0f27bc96-0c0d-4011-a836-3e4e6d5db5c3_isrForCrosstalkSources_998108_43.stderr parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b30e23b49241fa546fe1382471f4fa5b with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/c74debd2-c6ac-45fd-a391-63bf31c86fb6_isrForCrosstalkSources_998211_46.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/c74debd2-c6ac-45fd-a391-63bf31c86fb6_isrForCrosstalkSources_998211_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:57 proc_lsst.multi:146 [INFO] found job 17093100 in provider astro proc_lsst.multi INFO: found job 17093100 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:57 proc_lsst.multi:146 [INFO] found job 17093101 in provider astro proc_lsst.multi INFO: found job 17093101 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:48:58 proc_lsst.multi:146 [INFO] found job 17093102 in provider astro proc_lsst.multi INFO: found job 17093102 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:00 proc_lsst.multi:146 [INFO] found job 17093103 in provider astro proc_lsst.multi INFO: found job 17093103 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:00 proc_lsst.multi:146 [INFO] found job 17093104 in provider astro proc_lsst.multi INFO: found job 17093104 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:00 proc_lsst.multi:146 [INFO] found job 17093105 in provider astro proc_lsst.multi INFO: found job 17093105 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:00 proc_lsst.multi:146 [INFO] found job 17093106 in provider astro proc_lsst.multi INFO: found job 17093106 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:01 proc_lsst.multi:146 [INFO] found job 17093107 in provider astro proc_lsst.multi INFO: found job 17093107 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:01 proc_lsst.multi:146 [INFO] found job 17093108 in provider astro proc_lsst.multi INFO: found job 17093108 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:01 proc_lsst.multi:146 [INFO] found job 17093109 in provider astro proc_lsst.multi INFO: found job 17093109 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:01 proc_lsst.multi:146 [INFO] found job 17093110 in provider astro proc_lsst.multi INFO: found job 17093110 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:02 proc_lsst.multi:146 [INFO] found job 17093113 in provider astro proc_lsst.multi INFO: found job 17093113 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:02 proc_lsst.multi:146 [INFO] found job 17093114 in provider astro proc_lsst.multi INFO: found job 17093114 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:02 proc_lsst.multi:146 [INFO] found job 17093115 in provider astro proc_lsst.multi INFO: found job 17093115 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:02 proc_lsst.multi:146 [INFO] found job 17093116 in provider astro proc_lsst.multi INFO: found job 17093116 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093085,17093086,17093087,17093088,17093089,17093090,17093091,17093092,17093093,17093094,17093095,17093096,17093097,17093098,17093099,17093100,17093101,17093102,17093103,17093104,17093105,17093106,17093107,17093108,17093109,17093110,17093113,17093114,17093115,17093116' parsl.providers.slurm.slurm DEBUG: squeue returned 17093114 R 17093115 R 17093116 R 17093103 R 17093104 R 17093105 R 17093106 R 17093107 R 17093108 R 17093109 R 17093110 R 17093113 R 17093095 R 17093096 R 17093097 R 17093098 R 17093099 R 17093100 R 17093101 R 17093102 R 17093086 R 17093087 R 17093088 R 17093089 R 17093090 R 17093091 R 17093092 R 17093093 R 17093094 R 17093085 R parsl.providers.slurm.slurm DEBUG: Updating job 17093114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093085 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:03 proc_lsst.multi:146 [INFO] found job 17093117 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093117 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093117,17093118,17093119,17093120,17093121' parsl.providers.slurm.slurm DEBUG: squeue returned 17093117 R 17093118 R 17093119 R 17093120 R 17093121 R parsl.providers.slurm.slurm DEBUG: Updating job 17093117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093121 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:03 proc_lsst.multi:146 [INFO] found job 17093118 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093118 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093117,17093118,17093119,17093120,17093121' parsl.providers.slurm.slurm DEBUG: squeue returned 17093117 R 17093118 R 17093119 R 17093120 R 17093121 R parsl.providers.slurm.slurm DEBUG: Updating job 17093117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093121 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:03 proc_lsst.multi:146 [INFO] found job 17093119 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093119 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093117,17093118,17093119,17093120,17093121' parsl.providers.slurm.slurm DEBUG: squeue returned 17093117 R 17093118 R 17093119 R 17093120 R 17093121 R parsl.providers.slurm.slurm DEBUG: Updating job 17093117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093121 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:03 proc_lsst.multi:146 [INFO] found job 17093120 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093120 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093117,17093118,17093119,17093120,17093121' parsl.providers.slurm.slurm DEBUG: squeue returned 17093117 R 17093118 R 17093119 R 17093120 R 17093121 R parsl.providers.slurm.slurm DEBUG: Updating job 17093117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093121 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:03 proc_lsst.multi:146 [INFO] found job 17093121 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093121 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093117,17093118,17093119,17093120,17093121' parsl.providers.slurm.slurm DEBUG: squeue returned 17093117 R 17093118 R 17093119 R 17093120 R 17093121 R parsl.providers.slurm.slurm DEBUG: Updating job 17093117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093121 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:04 proc_lsst.multi:146 [INFO] found job 17093122 in provider escience_a40 proc_lsst.multi INFO: found job 17093122 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:04 proc_lsst.multi:146 [INFO] found job 17093123 in provider escience_a40 proc_lsst.multi INFO: found job 17093123 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:04 proc_lsst.multi:146 [INFO] found job 17093124 in provider escience_a40 proc_lsst.multi INFO: found job 17093124 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:04 proc_lsst.multi:146 [INFO] found job 17093125 in provider escience_a40 proc_lsst.multi INFO: found job 17093125 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:05 proc_lsst.multi:146 [INFO] found job 17093126 in provider escience_a40 proc_lsst.multi INFO: found job 17093126 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:05 proc_lsst.multi:146 [INFO] found job 17093127 in provider escience_a40 proc_lsst.multi INFO: found job 17093127 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:05 proc_lsst.multi:146 [INFO] found job 17093128 in provider escience_a40 proc_lsst.multi INFO: found job 17093128 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:05 proc_lsst.multi:146 [INFO] found job 17093129 in provider escience_a40 proc_lsst.multi INFO: found job 17093129 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:05 proc_lsst.multi:146 [INFO] found job 17093130 in provider escience_a40 proc_lsst.multi INFO: found job 17093130 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:06 proc_lsst.multi:146 [INFO] found job 17093132 in provider escience_a40 proc_lsst.multi INFO: found job 17093132 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:06 proc_lsst.multi:146 [INFO] found job 17093133 in provider escience_a40 proc_lsst.multi INFO: found job 17093133 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:06 proc_lsst.multi:146 [INFO] found job 17093135 in provider escience_a40 proc_lsst.multi INFO: found job 17093135 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:06 proc_lsst.multi:146 [INFO] found job 17093136 in provider escience_a40 proc_lsst.multi INFO: found job 17093136 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:06 proc_lsst.multi:146 [INFO] found job 17093138 in provider escience_a40 proc_lsst.multi INFO: found job 17093138 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:07 proc_lsst.multi:146 [INFO] found job 17093140 in provider escience_a40 proc_lsst.multi INFO: found job 17093140 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:07 proc_lsst.multi:146 [INFO] found job 17093142 in provider escience_a40 proc_lsst.multi INFO: found job 17093142 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:07 proc_lsst.multi:146 [INFO] found job 17093143 in provider escience_a40 proc_lsst.multi INFO: found job 17093143 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:07 proc_lsst.multi:146 [INFO] found job 17093145 in provider escience_a40 proc_lsst.multi INFO: found job 17093145 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:08 proc_lsst.multi:146 [INFO] found job 17093146 in provider escience_a40 proc_lsst.multi INFO: found job 17093146 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:08 proc_lsst.multi:146 [INFO] found job 17093147 in provider escience_a40 proc_lsst.multi INFO: found job 17093147 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093122,17093123,17093124,17093125,17093126,17093127,17093128,17093129,17093130,17093132,17093133,17093135,17093136,17093138,17093140,17093142,17093143,17093145,17093146,17093147' parsl.providers.slurm.slurm DEBUG: squeue returned 17093146 R 17093147 R 17093143 R 17093145 R 17093138 R 17093140 R 17093142 R 17093125 R 17093126 R 17093127 R 17093128 R 17093129 R 17093130 R 17093132 R 17093133 R 17093135 R 17093136 R 17093122 R 17093123 R 17093124 R parsl.providers.slurm.slurm DEBUG: Updating job 17093146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093124 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:08 proc_lsst.multi:146 [INFO] found job 17093149 in provider checkpoint proc_lsst.multi INFO: found job 17093149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:08 proc_lsst.multi:146 [INFO] found job 17093150 in provider checkpoint proc_lsst.multi INFO: found job 17093150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:08 proc_lsst.multi:146 [INFO] found job 17093151 in provider checkpoint proc_lsst.multi INFO: found job 17093151 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:09 proc_lsst.multi:146 [INFO] found job 17093153 in provider checkpoint proc_lsst.multi INFO: found job 17093153 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:09 proc_lsst.multi:146 [INFO] found job 17093154 in provider checkpoint proc_lsst.multi INFO: found job 17093154 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:09 proc_lsst.multi:146 [INFO] found job 17093155 in provider checkpoint proc_lsst.multi INFO: found job 17093155 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:09 proc_lsst.multi:146 [INFO] found job 17093157 in provider checkpoint proc_lsst.multi INFO: found job 17093157 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:10 proc_lsst.multi:146 [INFO] found job 17093158 in provider checkpoint proc_lsst.multi INFO: found job 17093158 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:10 proc_lsst.multi:146 [INFO] found job 17093159 in provider checkpoint proc_lsst.multi INFO: found job 17093159 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:10 proc_lsst.multi:146 [INFO] found job 17093160 in provider checkpoint proc_lsst.multi INFO: found job 17093160 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:10 proc_lsst.multi:146 [INFO] found job 17093161 in provider checkpoint proc_lsst.multi INFO: found job 17093161 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:11 proc_lsst.multi:146 [INFO] found job 17093162 in provider checkpoint proc_lsst.multi INFO: found job 17093162 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:11 proc_lsst.multi:146 [INFO] found job 17093163 in provider checkpoint proc_lsst.multi INFO: found job 17093163 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:11 proc_lsst.multi:146 [INFO] found job 17093164 in provider checkpoint proc_lsst.multi INFO: found job 17093164 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:11 proc_lsst.multi:146 [INFO] found job 17093165 in provider checkpoint proc_lsst.multi INFO: found job 17093165 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:11 proc_lsst.multi:146 [INFO] found job 17093166 in provider checkpoint proc_lsst.multi INFO: found job 17093166 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:12 proc_lsst.multi:146 [INFO] found job 17093167 in provider checkpoint proc_lsst.multi INFO: found job 17093167 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:12 proc_lsst.multi:146 [INFO] found job 17093168 in provider checkpoint proc_lsst.multi INFO: found job 17093168 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:12 proc_lsst.multi:146 [INFO] found job 17093169 in provider checkpoint proc_lsst.multi INFO: found job 17093169 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:12 proc_lsst.multi:146 [INFO] found job 17093170 in provider checkpoint proc_lsst.multi INFO: found job 17093170 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:13 proc_lsst.multi:146 [INFO] found job 17093171 in provider checkpoint proc_lsst.multi INFO: found job 17093171 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:13 proc_lsst.multi:146 [INFO] found job 17093172 in provider checkpoint proc_lsst.multi INFO: found job 17093172 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:13 proc_lsst.multi:146 [INFO] found job 17093173 in provider checkpoint proc_lsst.multi INFO: found job 17093173 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:13 proc_lsst.multi:146 [INFO] found job 17093174 in provider checkpoint proc_lsst.multi INFO: found job 17093174 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d56fb0b10c10d390dd7b4484c2806f3 with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/124b73cb-6b89-4b75-a822-9218c6095c18_isrForCrosstalkSources_998212_18.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/124b73cb-6b89-4b75-a822-9218c6095c18_isrForCrosstalkSources_998212_18.stderr parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a74ec417bfcea40587c9152b0ca3b3f7 with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/45057b27-06b0-464f-8ffb-14302533cd2e_isrForCrosstalkSources_998106_29.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/45057b27-06b0-464f-8ffb-14302533cd2e_isrForCrosstalkSources_998106_29.stderr parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de007bf191289503b4dd845d0249f1a6 with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/0a61cd6d-6b47-4d95-8978-def314e2db1b_isrForCrosstalkSources_998210_3.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/0a61cd6d-6b47-4d95-8978-def314e2db1b_isrForCrosstalkSources_998210_3.stderr parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 096d926f8a3319870edb75e546a30fbc with result from task 359 parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/e05343e7-1be3-4b67-9f67-8052b8c26446_isrForCrosstalkSources_998212_31.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/e05343e7-1be3-4b67-9f67-8052b8c26446_isrForCrosstalkSources_998212_31.stderr parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5ed68f41c73a5096200e2e7ef49cc11 with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/914fa5ad-44c8-4864-b6fa-c011bc43fc76_isrForCrosstalkSources_998211_35.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/914fa5ad-44c8-4864-b6fa-c011bc43fc76_isrForCrosstalkSources_998211_35.stderr parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2e2cc917e72cf08e4ee5774294ac73a with result from task 361 parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/9e41ada0-11a8-405b-9879-e8d55bfdb4ba_isrForCrosstalkSources_998210_19.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/9e41ada0-11a8-405b-9879-e8d55bfdb4ba_isrForCrosstalkSources_998210_19.stderr parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c93e83906c7466eb5cd2c01bf8cf09d5 with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/25ddb5ba-fa18-4190-9658-a960e0e1b31d_isrForCrosstalkSources_998210_25.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/25ddb5ba-fa18-4190-9658-a960e0e1b31d_isrForCrosstalkSources_998210_25.stderr parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dfb18a2b918ce3abce0419efe6c7cae with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/aafd10cb-7f0e-43ca-b1b6-c8b7c842512f_isrForCrosstalkSources_998108_6.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/aafd10cb-7f0e-43ca-b1b6-c8b7c842512f_isrForCrosstalkSources_998108_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:14 proc_lsst.multi:146 [INFO] found job 17093175 in provider checkpoint proc_lsst.multi INFO: found job 17093175 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:14 proc_lsst.multi:146 [INFO] found job 17093176 in provider checkpoint proc_lsst.multi INFO: found job 17093176 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:14 proc_lsst.multi:146 [INFO] found job 17093177 in provider checkpoint proc_lsst.multi INFO: found job 17093177 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:14 proc_lsst.multi:146 [INFO] found job 17093178 in provider checkpoint proc_lsst.multi INFO: found job 17093178 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:14 proc_lsst.multi:146 [INFO] found job 17093179 in provider checkpoint proc_lsst.multi INFO: found job 17093179 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:15 proc_lsst.multi:146 [INFO] found job 17093180 in provider checkpoint proc_lsst.multi INFO: found job 17093180 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:15 proc_lsst.multi:146 [INFO] found job 17093181 in provider checkpoint proc_lsst.multi INFO: found job 17093181 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a78cdfaede4bb8844f0f361f86c71de7 with result from task 328 parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/99de122a-fde2-4b3e-9fb3-87f66661bd03_isrForCrosstalkSources_998212_4.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/99de122a-fde2-4b3e-9fb3-87f66661bd03_isrForCrosstalkSources_998212_4.stderr parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5008b29fab2b6ece752fc29729de8f6d with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/31235152-ece2-4529-be7b-cf32375c45ca_isrForCrosstalkSources_998106_4.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/31235152-ece2-4529-be7b-cf32375c45ca_isrForCrosstalkSources_998106_4.stderr parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e9d82ec09c686e903155d1e5618d650 with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/064ea1fa-08b9-48b3-a09c-54921648e171_isrForCrosstalkSources_998210_8.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/064ea1fa-08b9-48b3-a09c-54921648e171_isrForCrosstalkSources_998210_8.stderr parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a9d1f967b4b810375212a525107f62f with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/8bd90400-9f77-46c6-a5f8-da10ecc5c8c5_isrForCrosstalkSources_998212_32.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/8bd90400-9f77-46c6-a5f8-da10ecc5c8c5_isrForCrosstalkSources_998212_32.stderr parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5093e491b31ac52bf741c6ba725a85bf with result from task 331 parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/d18f5946-2ab1-485f-a828-f25cf04e6bf9_isrForCrosstalkSources_998212_17.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/d18f5946-2ab1-485f-a828-f25cf04e6bf9_isrForCrosstalkSources_998212_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:15 proc_lsst.multi:146 [INFO] found job 17093182 in provider checkpoint proc_lsst.multi INFO: found job 17093182 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:15 proc_lsst.multi:146 [INFO] found job 17093183 in provider checkpoint proc_lsst.multi INFO: found job 17093183 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:16 proc_lsst.multi:146 [INFO] found job 17093184 in provider checkpoint proc_lsst.multi INFO: found job 17093184 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d657bcc6c2c9460c5d658f48b03497b7 with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/ea16de84-f56e-4829-9600-b783556aed93_isrForCrosstalkSources_998106_6.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/ea16de84-f56e-4829-9600-b783556aed93_isrForCrosstalkSources_998106_6.stderr parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63621e08e54d3cd8fa822be4a902f068 with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/eb0f1aa6-d52b-40d7-9b48-c8b7b2e3780c_isrForCrosstalkSources_998107_17.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/eb0f1aa6-d52b-40d7-9b48-c8b7b2e3780c_isrForCrosstalkSources_998107_17.stderr parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0ced0189010d2888d89fc72154903d9 with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/cc08f4fa-f04c-4ab7-b1a3-93b2528f612e_isrForCrosstalkSources_998106_7.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/cc08f4fa-f04c-4ab7-b1a3-93b2528f612e_isrForCrosstalkSources_998106_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:16 proc_lsst.multi:146 [INFO] found job 17093185 in provider checkpoint proc_lsst.multi INFO: found job 17093185 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Storing app cache entry c8d9166c81bb15b19a6bea6e69f02af9 with result from task 354 parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/e5446686-2ceb-4ee9-b64b-1cc6accba819_isrForCrosstalkSources_998107_8.stdout parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/e5446686-2ceb-4ee9-b64b-1cc6accba819_isrForCrosstalkSources_998107_8.stderr parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:16 proc_lsst.multi:146 [INFO] found job 17093186 in provider checkpoint parsl.dataflow.memoization DEBUG: Storing app cache entry 2605a29bb0cdf8a857b463775b770044 with result from task 345 proc_lsst.multi INFO: found job 17093186 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/00c33394-4454-4a6e-ad80-887be6122401_isrForCrosstalkSources_998108_51.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/00c33394-4454-4a6e-ad80-887be6122401_isrForCrosstalkSources_998108_51.stderr parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e137765d9cfd504bb32a2eedffcfcb57 with result from task 344 parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/bade858c-a2e2-4e6c-a471-b4b29936e3e0_isrForCrosstalkSources_998106_40.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/bade858c-a2e2-4e6c-a471-b4b29936e3e0_isrForCrosstalkSources_998106_40.stderr parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 956d4dd261d35a89046c21b32195e62d with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/0a305cf0-59c8-49ae-8e65-1bf66e91da17_isrForCrosstalkSources_998106_56.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/0a305cf0-59c8-49ae-8e65-1bf66e91da17_isrForCrosstalkSources_998106_56.stderr parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 331616c74750d23eb458553c1acfc05f with result from task 341 parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/3627f4ef-63cf-4ffb-9228-67eef59d2f7d_isrForCrosstalkSources_998211_13.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/3627f4ef-63cf-4ffb-9228-67eef59d2f7d_isrForCrosstalkSources_998211_13.stderr parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 081a34dd7bcb3ed2f17804237f7ba9f6 with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/226c60a5-7061-4c63-a2b5-2ed1da64fc7a_isrForCrosstalkSources_998106_18.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/226c60a5-7061-4c63-a2b5-2ed1da64fc7a_isrForCrosstalkSources_998106_18.stderr parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9eacb3c6ad06efacf41cf1e7352eb9a3 with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/82358c58-29f5-4600-b362-605f8dc2df43_isrForCrosstalkSources_998107_33.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/82358c58-29f5-4600-b362-605f8dc2df43_isrForCrosstalkSources_998107_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING 2024-03-15 22:49:16 proc_lsst.multi:146 [INFO] found job 17093187 in provider checkpoint proc_lsst.multi INFO: found job 17093187 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17093149,17093150,17093151,17093153,17093154,17093155,17093157,17093158,17093159,17093160,17093161,17093162,17093163,17093164,17093165,17093166,17093167,17093168,17093169,17093170,17093171,17093172,17093173,17093174,17093175,17093176,17093177,17093178,17093179,17093180,17093181,17093182,17093183,17093184,17093185,17093186,17093187' parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5946f984c4191e5b1ecaebfb23f997c0 with result from task 343 parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/98a5a088-c14a-4789-afbd-37857a442aff_isrForCrosstalkSources_998106_2.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998106/98a5a088-c14a-4789-afbd-37857a442aff_isrForCrosstalkSources_998106_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17093161 PD 17093162 PD 17093163 PD 17093164 PD 17093165 PD 17093166 PD 17093167 PD 17093168 PD 17093169 PD 17093170 PD 17093171 PD 17093172 PD 17093173 PD 17093174 PD 17093175 PD 17093176 PD 17093177 PD 17093178 PD 17093179 PD 17093180 PD 17093181 PD 17093182 PD 17093183 PD 17093184 PD 17093185 PD 17093186 PD 17093187 PD 17093149 R 17093150 R 17093151 R 17093153 R 17093154 R 17093155 R 17093157 R 17093158 R 17093159 R 17093160 R parsl.providers.slurm.slurm DEBUG: Updating job 17093161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17093149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17093160 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 22 parsl.dataflow.strategy DEBUG: Executor multi has 22 active tasks, 66/27 running/pending blocks, and 137 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22773982342992 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 22 parsl.dataflow.strategy DEBUG: Executor multi has 22 active tasks, 66/27 running/pending blocks, and 137 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22773982342992 parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bc3b576acec5256116f94f5789ec417 with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/6856d0dd-c33a-4678-b768-b17f2356440d_isrForCrosstalkSources_998212_44.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/6856d0dd-c33a-4678-b768-b17f2356440d_isrForCrosstalkSources_998212_44.stderr parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 995a28bb3cbefa09952a9410fdfc904f with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/0ea6c560-6265-421b-b782-0baee2ef971a_isrForCrosstalkSources_998108_19.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/0ea6c560-6265-421b-b782-0baee2ef971a_isrForCrosstalkSources_998108_19.stderr parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4529a30fe7b4ece4051df4af4c1d3eed with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/efaed595-9100-40cd-9f92-b372f0c2c93b_isrForCrosstalkSources_998211_25.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/efaed595-9100-40cd-9f92-b372f0c2c93b_isrForCrosstalkSources_998211_25.stderr parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f1ea70a0b3790c7d2f6b2ac13c7e83e with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/350f78d5-2c26-4c23-88fe-ef26bddbfe6a_isrForCrosstalkSources_998107_1.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/350f78d5-2c26-4c23-88fe-ef26bddbfe6a_isrForCrosstalkSources_998107_1.stderr parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc465582266b157a2132e49834fe336c with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/cd2a7d75-e55d-4321-9ab6-933742313e25_isrForCrosstalkSources_998107_37.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/cd2a7d75-e55d-4321-9ab6-933742313e25_isrForCrosstalkSources_998107_37.stderr parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 522102290bd4f680f6f4a1d98629b69c with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/da3ec7c1-e4ec-4849-835a-ad93e519f56d_isrForCrosstalkSources_998108_15.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/da3ec7c1-e4ec-4849-835a-ad93e519f56d_isrForCrosstalkSources_998108_15.stderr parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68731608689bbf862b24eb5a0eca12c1 with result from task 367 parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/b151d595-4b74-444f-a567-03ca3718c542_isrForCrosstalkSources_998212_16.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/b151d595-4b74-444f-a567-03ca3718c542_isrForCrosstalkSources_998212_16.stderr parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dc6882a4fabb853997f125ca6b270f3 with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/bf94558f-0df6-4680-bc2a-df8e3f6ca360_isrForCrosstalkSources_998211_30.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/bf94558f-0df6-4680-bc2a-df8e3f6ca360_isrForCrosstalkSources_998211_30.stderr parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b5650894f6ec96e74368e33ef32d214 with result from task 346 parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/a68d116f-ccb6-495c-b73c-b8095ca66c68_isrForCrosstalkSources_998211_29.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/a68d116f-ccb6-495c-b73c-b8095ca66c68_isrForCrosstalkSources_998211_29.stderr parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d187295fd1378fc390a722c63c5240a8 with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/f4751bb5-8a9b-4b47-a2d2-bd4c204c7d79_isrForCrosstalkSources_998211_54.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/f4751bb5-8a9b-4b47-a2d2-bd4c204c7d79_isrForCrosstalkSources_998211_54.stderr parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd1e917ed37a5dc5db1204c92790eb19 with result from task 347 parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/ff96791f-38ed-4c56-b6e5-39a92cd5148d_isrForCrosstalkSources_998210_30.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/ff96791f-38ed-4c56-b6e5-39a92cd5148d_isrForCrosstalkSources_998210_30.stderr parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f9071b175986aeddaf727865664796a with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/8393a5a1-852e-467c-bc49-58e33cba3b83_isrForCrosstalkSources_998210_29.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/8393a5a1-852e-467c-bc49-58e33cba3b83_isrForCrosstalkSources_998210_29.stderr parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6490151bfd681b1f9e72c4e550620ba6 with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/39f8f9a9-df4e-496f-b856-8c7d9712ac15_isrForCrosstalkSources_998210_53.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/39f8f9a9-df4e-496f-b856-8c7d9712ac15_isrForCrosstalkSources_998210_53.stderr parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef136381192bf14615c56aacb115bf88 with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/0427bb26-24e1-4207-8d8a-7eb00a034749_isrForCrosstalkSources_998108_59.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998108/0427bb26-24e1-4207-8d8a-7eb00a034749_isrForCrosstalkSources_998108_59.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 66/27 running/pending blocks, and 137 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22773982342992 parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47f620b98b22b459f2f35e61af1ed671 with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/931604c6-088a-4aaa-96ef-eb8ac38009c8_isrForCrosstalkSources_998212_22.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/931604c6-088a-4aaa-96ef-eb8ac38009c8_isrForCrosstalkSources_998212_22.stderr parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c69ba5a9a1f8ea86562591401f7a946f with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/a9cb2021-aded-4553-a8e7-7b74fbf0648e_isrForCrosstalkSources_998107_48.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/a9cb2021-aded-4553-a8e7-7b74fbf0648e_isrForCrosstalkSources_998107_48.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b201125061647c7735bb161aaab1bf8 with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/90ef2c7b-f237-4f46-8bfb-4dd671ca050c_isrForCrosstalkSources_998210_50.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/90ef2c7b-f237-4f46-8bfb-4dd671ca050c_isrForCrosstalkSources_998210_50.stderr parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 970061fdb4198afd89bcc1e0b3acce27 with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/ec2d1151-8cad-45b8-ba52-bd7c8c240135_isrForCrosstalkSources_998211_59.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998211/ec2d1151-8cad-45b8-ba52-bd7c8c240135_isrForCrosstalkSources_998211_59.stderr parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77e18d096d4df9702e197e1d29c9deba with result from task 335 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/bffe803b-6f78-45d0-b3a8-1360d291a884_isrForCrosstalkSources_998107_24.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998107/bffe803b-6f78-45d0-b3a8-1360d291a884_isrForCrosstalkSources_998107_24.stderr parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81125aa8ac0ca1668e03fd0b8bd733e5 with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/3c629a09-3686-4e42-89c4-1e55520d6512_isrForCrosstalkSources_998210_10.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998210/3c629a09-3686-4e42-89c4-1e55520d6512_isrForCrosstalkSources_998210_10.stderr parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7eb74515d7337d63ee72e882877c8244 with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/052e7028-53d4-4ac8-b3e8-7edd19db28e9_isrForCrosstalkSources_998212_52.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/052e7028-53d4-4ac8-b3e8-7edd19db28e9_isrForCrosstalkSources_998212_52.stderr parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 086f9501efe4df7b20c91b9a0702bed7 with result from task 338 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 372 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Tasks in state States.failed: 0 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/ad8c7e95-9e3d-4e60-ae5d-6671642ec429_isrForCrosstalkSources_998212_47.stdout parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/submit/DEEP/20210516/A0g/science#step0/20240315T224631Z/logs/isrForCrosstalkSources/998212/ad8c7e95-9e3d-4e60-ae5d-6671642ec429_isrForCrosstalkSources_998212_47.stderr parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=93, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 66 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0aef6afea18a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0aef6afea18a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 489c37e64759 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 489c37e64759 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d314b047f90b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d314b047f90b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 509cac9292c2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 509cac9292c2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 867ebd192b20 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 867ebd192b20 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4b933fc2aa78 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4b933fc2aa78 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5f0dc313172d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5f0dc313172d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4ef0bf734407 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4ef0bf734407 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 419a91f6f820 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 419a91f6f820 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 181adbcb1e1d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 181adbcb1e1d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ee945cdef09d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ee945cdef09d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d4b10f873ec1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d4b10f873ec1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b1fbc3c21edb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b1fbc3c21edb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 938e5aea031b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 938e5aea031b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a63c559b07db parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a63c559b07db parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2e38f7e0738d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2e38f7e0738d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4de7f7ddaf12 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4de7f7ddaf12 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 17f46ac5dd6c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 17f46ac5dd6c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ff2549ccbab0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ff2549ccbab0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 12e0a3b65b67 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 12e0a3b65b67 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 726b9a4be003 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 726b9a4be003 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4083ed8056b8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4083ed8056b8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1926a9df73fa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1926a9df73fa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bab3cce4a9a5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bab3cce4a9a5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: febdaa4af783 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: febdaa4af783 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: abf272665f1f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: abf272665f1f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 156044f8f270 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 156044f8f270 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 75e12d878665 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 75e12d878665 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7741d0a71036 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7741d0a71036 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ef83a557162b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ef83a557162b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0dce090aa4f5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0dce090aa4f5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ee7810425d2e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ee7810425d2e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f76d84a20e86 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f76d84a20e86 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 110523fa4fcc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 110523fa4fcc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e23da1bdb187 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e23da1bdb187 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b7d7157a8b02 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b7d7157a8b02 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a36f36b89fde parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a36f36b89fde parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d7eeee8c32ff parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d7eeee8c32ff parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2878651517f8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2878651517f8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ffe79acd9850 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ffe79acd9850 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2c2356cf44bc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2c2356cf44bc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4ceb26727899 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4ceb26727899 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fe501cec812c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fe501cec812c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4c0842c10268 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4c0842c10268 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d11541ee23c5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d11541ee23c5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 25d5fffb4bb7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 25d5fffb4bb7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 580c853bc40f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 580c853bc40f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dd77c260bafc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dd77c260bafc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b6f75a0c8211 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b6f75a0c8211 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 75289bad9621 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 75289bad9621 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5a29bafa9c72 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5a29bafa9c72 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 25201b8a8ce4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 25201b8a8ce4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1fb5a199de7c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1fb5a199de7c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1f1f63757203 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1f1f63757203 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9c53de3b8b48 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9c53de3b8b48 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cb1b49dcb58f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cb1b49dcb58f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c3ffbf6622d0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c3ffbf6622d0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 09b7ea0815cb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 09b7ea0815cb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a1bb7dc8d537 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a1bb7dc8d537 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 494ef4f494fe parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 494ef4f494fe parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 01b2d2512860 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 01b2d2512860 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f3839f4d85de parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f3839f4d85de parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c1b3119dcb79 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c1b3119dcb79 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5e914382eb9a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5e914382eb9a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0d17a2a775bd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0d17a2a775bd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5723e9679c0f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5723e9679c0f 2024-03-15 22:49:23 proc_lsst.multi:146 [INFO] found job 17093159 in provider checkpoint proc_lsst.multi INFO: found job 17093159 in provider checkpoint 2024-03-15 22:49:23 proc_lsst.multi:201 [INFO] cancelling 17093159 on provider checkpoint proc_lsst.multi INFO: cancelling 17093159 on provider checkpoint 2024-03-15 22:49:23 proc_lsst.multi:146 [INFO] found job 17093154 in provider checkpoint proc_lsst.multi INFO: found job 17093154 in provider checkpoint 2024-03-15 22:49:23 proc_lsst.multi:201 [INFO] cancelling 17093154 on provider checkpoint proc_lsst.multi INFO: cancelling 17093154 on provider checkpoint 2024-03-15 22:49:23 proc_lsst.multi:146 [INFO] found job 17093153 in provider checkpoint proc_lsst.multi INFO: found job 17093153 in provider checkpoint 2024-03-15 22:49:23 proc_lsst.multi:201 [INFO] cancelling 17093153 on provider checkpoint proc_lsst.multi INFO: cancelling 17093153 on provider checkpoint 2024-03-15 22:49:23 proc_lsst.multi:146 [INFO] found job 17093160 in provider checkpoint proc_lsst.multi INFO: found job 17093160 in provider checkpoint 2024-03-15 22:49:23 proc_lsst.multi:201 [INFO] cancelling 17093160 on provider checkpoint proc_lsst.multi INFO: cancelling 17093160 on provider checkpoint 2024-03-15 22:49:23 proc_lsst.multi:146 [INFO] found job 17093151 in provider checkpoint proc_lsst.multi INFO: found job 17093151 in provider checkpoint 2024-03-15 22:49:23 proc_lsst.multi:201 [INFO] cancelling 17093151 on provider checkpoint proc_lsst.multi INFO: cancelling 17093151 on provider checkpoint 2024-03-15 22:49:24 proc_lsst.multi:146 [INFO] found job 17093149 in provider checkpoint proc_lsst.multi INFO: found job 17093149 in provider checkpoint 2024-03-15 22:49:24 proc_lsst.multi:201 [INFO] cancelling 17093149 on provider checkpoint proc_lsst.multi INFO: cancelling 17093149 on provider checkpoint 2024-03-15 22:49:24 proc_lsst.multi:146 [INFO] found job 36385 in provider local proc_lsst.multi INFO: found job 36385 in provider local 2024-03-15 22:49:24 proc_lsst.multi:201 [INFO] cancelling 36385 on provider local proc_lsst.multi INFO: cancelling 36385 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 36385 2024-03-15 22:49:24 proc_lsst.multi:146 [INFO] found job 17093094 in provider astro proc_lsst.multi INFO: found job 17093094 in provider astro 2024-03-15 22:49:24 proc_lsst.multi:201 [INFO] cancelling 17093094 on provider astro proc_lsst.multi INFO: cancelling 17093094 on provider astro 2024-03-15 22:49:24 proc_lsst.multi:146 [INFO] found job 17093100 in provider astro proc_lsst.multi INFO: found job 17093100 in provider astro 2024-03-15 22:49:24 proc_lsst.multi:201 [INFO] cancelling 17093100 on provider astro proc_lsst.multi INFO: cancelling 17093100 on provider astro 2024-03-15 22:49:24 proc_lsst.multi:146 [INFO] found job 17093103 in provider astro proc_lsst.multi INFO: found job 17093103 in provider astro 2024-03-15 22:49:24 proc_lsst.multi:201 [INFO] cancelling 17093103 on provider astro proc_lsst.multi INFO: cancelling 17093103 on provider astro 2024-03-15 22:49:24 proc_lsst.multi:146 [INFO] found job 17093097 in provider astro proc_lsst.multi INFO: found job 17093097 in provider astro 2024-03-15 22:49:24 proc_lsst.multi:201 [INFO] cancelling 17093097 on provider astro proc_lsst.multi INFO: cancelling 17093097 on provider astro 2024-03-15 22:49:25 proc_lsst.multi:146 [INFO] found job 17093102 in provider astro proc_lsst.multi INFO: found job 17093102 in provider astro 2024-03-15 22:49:25 proc_lsst.multi:201 [INFO] cancelling 17093102 on provider astro proc_lsst.multi INFO: cancelling 17093102 on provider astro 2024-03-15 22:49:25 proc_lsst.multi:146 [INFO] found job 17093088 in provider astro proc_lsst.multi INFO: found job 17093088 in provider astro 2024-03-15 22:49:25 proc_lsst.multi:201 [INFO] cancelling 17093088 on provider astro proc_lsst.multi INFO: cancelling 17093088 on provider astro 2024-03-15 22:49:25 proc_lsst.multi:146 [INFO] found job 17093087 in provider astro proc_lsst.multi INFO: found job 17093087 in provider astro 2024-03-15 22:49:25 proc_lsst.multi:201 [INFO] cancelling 17093087 on provider astro proc_lsst.multi INFO: cancelling 17093087 on provider astro 2024-03-15 22:49:25 proc_lsst.multi:146 [INFO] found job 17093089 in provider astro proc_lsst.multi INFO: found job 17093089 in provider astro 2024-03-15 22:49:25 proc_lsst.multi:201 [INFO] cancelling 17093089 on provider astro proc_lsst.multi INFO: cancelling 17093089 on provider astro 2024-03-15 22:49:25 proc_lsst.multi:146 [INFO] found job 17093090 in provider astro proc_lsst.multi INFO: found job 17093090 in provider astro 2024-03-15 22:49:25 proc_lsst.multi:201 [INFO] cancelling 17093090 on provider astro proc_lsst.multi INFO: cancelling 17093090 on provider astro 2024-03-15 22:49:25 proc_lsst.multi:146 [INFO] found job 17093091 in provider astro proc_lsst.multi INFO: found job 17093091 in provider astro 2024-03-15 22:49:25 proc_lsst.multi:201 [INFO] cancelling 17093091 on provider astro proc_lsst.multi INFO: cancelling 17093091 on provider astro 2024-03-15 22:49:26 proc_lsst.multi:146 [INFO] found job 17093096 in provider astro proc_lsst.multi INFO: found job 17093096 in provider astro 2024-03-15 22:49:26 proc_lsst.multi:201 [INFO] cancelling 17093096 on provider astro proc_lsst.multi INFO: cancelling 17093096 on provider astro 2024-03-15 22:49:26 proc_lsst.multi:146 [INFO] found job 17093099 in provider astro proc_lsst.multi INFO: found job 17093099 in provider astro 2024-03-15 22:49:26 proc_lsst.multi:201 [INFO] cancelling 17093099 on provider astro proc_lsst.multi INFO: cancelling 17093099 on provider astro 2024-03-15 22:49:26 proc_lsst.multi:146 [INFO] found job 17093093 in provider astro proc_lsst.multi INFO: found job 17093093 in provider astro 2024-03-15 22:49:26 proc_lsst.multi:201 [INFO] cancelling 17093093 on provider astro proc_lsst.multi INFO: cancelling 17093093 on provider astro 2024-03-15 22:49:26 proc_lsst.multi:146 [INFO] found job 17093098 in provider astro proc_lsst.multi INFO: found job 17093098 in provider astro 2024-03-15 22:49:26 proc_lsst.multi:201 [INFO] cancelling 17093098 on provider astro proc_lsst.multi INFO: cancelling 17093098 on provider astro 2024-03-15 22:49:26 proc_lsst.multi:146 [INFO] found job 17093101 in provider astro proc_lsst.multi INFO: found job 17093101 in provider astro 2024-03-15 22:49:26 proc_lsst.multi:201 [INFO] cancelling 17093101 on provider astro proc_lsst.multi INFO: cancelling 17093101 on provider astro 2024-03-15 22:49:26 proc_lsst.multi:146 [INFO] found job 17093086 in provider astro proc_lsst.multi INFO: found job 17093086 in provider astro 2024-03-15 22:49:26 proc_lsst.multi:201 [INFO] cancelling 17093086 on provider astro proc_lsst.multi INFO: cancelling 17093086 on provider astro 2024-03-15 22:49:27 proc_lsst.multi:146 [INFO] found job 17093095 in provider astro proc_lsst.multi INFO: found job 17093095 in provider astro 2024-03-15 22:49:27 proc_lsst.multi:201 [INFO] cancelling 17093095 on provider astro proc_lsst.multi INFO: cancelling 17093095 on provider astro 2024-03-15 22:49:27 proc_lsst.multi:146 [INFO] found job 17093085 in provider astro proc_lsst.multi INFO: found job 17093085 in provider astro 2024-03-15 22:49:27 proc_lsst.multi:201 [INFO] cancelling 17093085 on provider astro proc_lsst.multi INFO: cancelling 17093085 on provider astro 2024-03-15 22:49:27 proc_lsst.multi:146 [INFO] found job 17093092 in provider astro proc_lsst.multi INFO: found job 17093092 in provider astro 2024-03-15 22:49:27 proc_lsst.multi:201 [INFO] cancelling 17093092 on provider astro proc_lsst.multi INFO: cancelling 17093092 on provider astro 2024-03-15 22:49:27 proc_lsst.multi:146 [INFO] found job 17093115 in provider astro proc_lsst.multi INFO: found job 17093115 in provider astro 2024-03-15 22:49:27 proc_lsst.multi:201 [INFO] cancelling 17093115 on provider astro proc_lsst.multi INFO: cancelling 17093115 on provider astro 2024-03-15 22:49:27 proc_lsst.multi:146 [INFO] found job 17093150 in provider checkpoint proc_lsst.multi INFO: found job 17093150 in provider checkpoint 2024-03-15 22:49:27 proc_lsst.multi:201 [INFO] cancelling 17093150 on provider checkpoint proc_lsst.multi INFO: cancelling 17093150 on provider checkpoint 2024-03-15 22:49:27 proc_lsst.multi:146 [INFO] found job 17093107 in provider astro proc_lsst.multi INFO: found job 17093107 in provider astro 2024-03-15 22:49:27 proc_lsst.multi:201 [INFO] cancelling 17093107 on provider astro proc_lsst.multi INFO: cancelling 17093107 on provider astro 2024-03-15 22:49:28 proc_lsst.multi:146 [INFO] found job 17093116 in provider astro proc_lsst.multi INFO: found job 17093116 in provider astro 2024-03-15 22:49:28 proc_lsst.multi:201 [INFO] cancelling 17093116 on provider astro proc_lsst.multi INFO: cancelling 17093116 on provider astro 2024-03-15 22:49:28 proc_lsst.multi:146 [INFO] found job 17093110 in provider astro proc_lsst.multi INFO: found job 17093110 in provider astro 2024-03-15 22:49:28 proc_lsst.multi:201 [INFO] cancelling 17093110 on provider astro proc_lsst.multi INFO: cancelling 17093110 on provider astro 2024-03-15 22:49:28 proc_lsst.multi:146 [INFO] found job 17093118 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093118 in provider escience_rtx6k 2024-03-15 22:49:28 proc_lsst.multi:201 [INFO] cancelling 17093118 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17093118 on provider escience_rtx6k 2024-03-15 22:49:28 proc_lsst.multi:146 [INFO] found job 17093109 in provider astro proc_lsst.multi INFO: found job 17093109 in provider astro 2024-03-15 22:49:28 proc_lsst.multi:201 [INFO] cancelling 17093109 on provider astro proc_lsst.multi INFO: cancelling 17093109 on provider astro 2024-03-15 22:49:28 proc_lsst.multi:146 [INFO] found job 17093121 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093121 in provider escience_rtx6k 2024-03-15 22:49:28 proc_lsst.multi:201 [INFO] cancelling 17093121 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17093121 on provider escience_rtx6k 2024-03-15 22:49:28 proc_lsst.multi:146 [INFO] found job 17093105 in provider astro proc_lsst.multi INFO: found job 17093105 in provider astro 2024-03-15 22:49:28 proc_lsst.multi:201 [INFO] cancelling 17093105 on provider astro proc_lsst.multi INFO: cancelling 17093105 on provider astro 2024-03-15 22:49:29 proc_lsst.multi:146 [INFO] found job 17093119 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093119 in provider escience_rtx6k 2024-03-15 22:49:29 proc_lsst.multi:201 [INFO] cancelling 17093119 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17093119 on provider escience_rtx6k 2024-03-15 22:49:29 proc_lsst.multi:146 [INFO] found job 17093113 in provider astro proc_lsst.multi INFO: found job 17093113 in provider astro 2024-03-15 22:49:29 proc_lsst.multi:201 [INFO] cancelling 17093113 on provider astro proc_lsst.multi INFO: cancelling 17093113 on provider astro 2024-03-15 22:49:29 proc_lsst.multi:146 [INFO] found job 17093117 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093117 in provider escience_rtx6k 2024-03-15 22:49:29 proc_lsst.multi:201 [INFO] cancelling 17093117 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17093117 on provider escience_rtx6k 2024-03-15 22:49:29 proc_lsst.multi:146 [INFO] found job 17093124 in provider escience_a40 proc_lsst.multi INFO: found job 17093124 in provider escience_a40 2024-03-15 22:49:29 proc_lsst.multi:201 [INFO] cancelling 17093124 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093124 on provider escience_a40 2024-03-15 22:49:29 proc_lsst.multi:146 [INFO] found job 17093127 in provider escience_a40 proc_lsst.multi INFO: found job 17093127 in provider escience_a40 2024-03-15 22:49:29 proc_lsst.multi:201 [INFO] cancelling 17093127 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093127 on provider escience_a40 2024-03-15 22:49:29 proc_lsst.multi:146 [INFO] found job 17093132 in provider escience_a40 proc_lsst.multi INFO: found job 17093132 in provider escience_a40 2024-03-15 22:49:29 proc_lsst.multi:201 [INFO] cancelling 17093132 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093132 on provider escience_a40 2024-03-15 22:49:30 proc_lsst.multi:146 [INFO] found job 17093120 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093120 in provider escience_rtx6k 2024-03-15 22:49:30 proc_lsst.multi:201 [INFO] cancelling 17093120 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17093120 on provider escience_rtx6k 2024-03-15 22:49:30 proc_lsst.multi:146 [INFO] found job 17093129 in provider escience_a40 proc_lsst.multi INFO: found job 17093129 in provider escience_a40 2024-03-15 22:49:30 proc_lsst.multi:201 [INFO] cancelling 17093129 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093129 on provider escience_a40 2024-03-15 22:49:30 proc_lsst.multi:146 [INFO] found job 17093155 in provider checkpoint proc_lsst.multi INFO: found job 17093155 in provider checkpoint 2024-03-15 22:49:30 proc_lsst.multi:201 [INFO] cancelling 17093155 on provider checkpoint proc_lsst.multi INFO: cancelling 17093155 on provider checkpoint 2024-03-15 22:49:30 proc_lsst.multi:146 [INFO] found job 17093125 in provider escience_a40 proc_lsst.multi INFO: found job 17093125 in provider escience_a40 2024-03-15 22:49:30 proc_lsst.multi:201 [INFO] cancelling 17093125 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093125 on provider escience_a40 2024-03-15 22:49:32 proc_lsst.multi:146 [INFO] found job 17093108 in provider astro proc_lsst.multi INFO: found job 17093108 in provider astro 2024-03-15 22:49:32 proc_lsst.multi:201 [INFO] cancelling 17093108 on provider astro proc_lsst.multi INFO: cancelling 17093108 on provider astro 2024-03-15 22:49:33 proc_lsst.multi:146 [INFO] found job 17093136 in provider escience_a40 proc_lsst.multi INFO: found job 17093136 in provider escience_a40 2024-03-15 22:49:33 proc_lsst.multi:201 [INFO] cancelling 17093136 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093136 on provider escience_a40 2024-03-15 22:49:35 proc_lsst.multi:146 [INFO] found job 17093106 in provider astro proc_lsst.multi INFO: found job 17093106 in provider astro 2024-03-15 22:49:35 proc_lsst.multi:201 [INFO] cancelling 17093106 on provider astro proc_lsst.multi INFO: cancelling 17093106 on provider astro 2024-03-15 22:49:35 proc_lsst.multi:146 [INFO] found job 17093140 in provider escience_a40 proc_lsst.multi INFO: found job 17093140 in provider escience_a40 2024-03-15 22:49:35 proc_lsst.multi:201 [INFO] cancelling 17093140 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093140 on provider escience_a40 2024-03-15 22:49:36 proc_lsst.multi:146 [INFO] found job 17093114 in provider astro proc_lsst.multi INFO: found job 17093114 in provider astro 2024-03-15 22:49:36 proc_lsst.multi:201 [INFO] cancelling 17093114 on provider astro proc_lsst.multi INFO: cancelling 17093114 on provider astro 2024-03-15 22:49:36 proc_lsst.multi:146 [INFO] found job 17093126 in provider escience_a40 proc_lsst.multi INFO: found job 17093126 in provider escience_a40 2024-03-15 22:49:36 proc_lsst.multi:201 [INFO] cancelling 17093126 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093126 on provider escience_a40 2024-03-15 22:49:36 proc_lsst.multi:146 [INFO] found job 17093130 in provider escience_a40 proc_lsst.multi INFO: found job 17093130 in provider escience_a40 2024-03-15 22:49:36 proc_lsst.multi:201 [INFO] cancelling 17093130 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093130 on provider escience_a40 2024-03-15 22:49:36 proc_lsst.multi:146 [INFO] found job 17093135 in provider escience_a40 proc_lsst.multi INFO: found job 17093135 in provider escience_a40 2024-03-15 22:49:36 proc_lsst.multi:201 [INFO] cancelling 17093135 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093135 on provider escience_a40 2024-03-15 22:49:37 proc_lsst.multi:146 [INFO] found job 17093123 in provider escience_a40 proc_lsst.multi INFO: found job 17093123 in provider escience_a40 2024-03-15 22:49:37 proc_lsst.multi:201 [INFO] cancelling 17093123 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093123 on provider escience_a40 2024-03-15 22:49:37 proc_lsst.multi:146 [INFO] found job 17093145 in provider escience_a40 proc_lsst.multi INFO: found job 17093145 in provider escience_a40 2024-03-15 22:49:37 proc_lsst.multi:201 [INFO] cancelling 17093145 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093145 on provider escience_a40 2024-03-15 22:49:37 proc_lsst.multi:146 [INFO] found job 17093133 in provider escience_a40 proc_lsst.multi INFO: found job 17093133 in provider escience_a40 2024-03-15 22:49:37 proc_lsst.multi:201 [INFO] cancelling 17093133 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093133 on provider escience_a40 2024-03-15 22:49:38 proc_lsst.multi:146 [INFO] found job 17093138 in provider escience_a40 proc_lsst.multi INFO: found job 17093138 in provider escience_a40 2024-03-15 22:49:38 proc_lsst.multi:201 [INFO] cancelling 17093138 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093138 on provider escience_a40 2024-03-15 22:49:38 proc_lsst.multi:146 [INFO] found job 17093122 in provider escience_a40 proc_lsst.multi INFO: found job 17093122 in provider escience_a40 2024-03-15 22:49:38 proc_lsst.multi:201 [INFO] cancelling 17093122 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093122 on provider escience_a40 2024-03-15 22:49:38 proc_lsst.multi:146 [INFO] found job 17093142 in provider escience_a40 proc_lsst.multi INFO: found job 17093142 in provider escience_a40 2024-03-15 22:49:38 proc_lsst.multi:201 [INFO] cancelling 17093142 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093142 on provider escience_a40 2024-03-15 22:49:38 proc_lsst.multi:146 [INFO] found job 17093128 in provider escience_a40 proc_lsst.multi INFO: found job 17093128 in provider escience_a40 2024-03-15 22:49:38 proc_lsst.multi:201 [INFO] cancelling 17093128 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093128 on provider escience_a40 2024-03-15 22:49:39 proc_lsst.multi:146 [INFO] found job 17093143 in provider escience_a40 proc_lsst.multi INFO: found job 17093143 in provider escience_a40 2024-03-15 22:49:39 proc_lsst.multi:201 [INFO] cancelling 17093143 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093143 on provider escience_a40 2024-03-15 22:49:39 proc_lsst.multi:146 [INFO] found job 17093147 in provider escience_a40 proc_lsst.multi INFO: found job 17093147 in provider escience_a40 2024-03-15 22:49:39 proc_lsst.multi:201 [INFO] cancelling 17093147 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093147 on provider escience_a40 2024-03-15 22:49:39 proc_lsst.multi:146 [INFO] found job 17093146 in provider escience_a40 proc_lsst.multi INFO: found job 17093146 in provider escience_a40 2024-03-15 22:49:39 proc_lsst.multi:201 [INFO] cancelling 17093146 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093146 on provider escience_a40 2024-03-15 22:49:39 proc_lsst.multi:146 [INFO] found job 17093104 in provider astro proc_lsst.multi INFO: found job 17093104 in provider astro 2024-03-15 22:49:39 proc_lsst.multi:201 [INFO] cancelling 17093104 on provider astro proc_lsst.multi INFO: cancelling 17093104 on provider astro 2024-03-15 22:49:39 proc_lsst.multi:146 [INFO] found job 17093157 in provider checkpoint proc_lsst.multi INFO: found job 17093157 in provider checkpoint 2024-03-15 22:49:39 proc_lsst.multi:201 [INFO] cancelling 17093157 on provider checkpoint proc_lsst.multi INFO: cancelling 17093157 on provider checkpoint 2024-03-15 22:49:40 proc_lsst.multi:146 [INFO] found job 17093158 in provider checkpoint proc_lsst.multi INFO: found job 17093158 in provider checkpoint 2024-03-15 22:49:40 proc_lsst.multi:201 [INFO] cancelling 17093158 on provider checkpoint proc_lsst.multi INFO: cancelling 17093158 on provider checkpoint parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-15 22:49:40 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-15 22:49:40 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17093149', '17093150', '17093151', '17093153', '17093154', '17093155', '17093157', '17093158', '17093159', '17093160', '17093161', '17093162', '17093163', '17093164', '17093165', '17093166', '17093167', '17093168', '17093169', '17093170', '17093171', '17093172', '17093173', '17093174', '17093175', '17093176', '17093177', '17093178', '17093179', '17093180', '17093181', '17093182', '17093183', '17093184', '17093185', '17093186', '17093187', '17093085', '17093086', '17093087', '17093088', '17093089', '17093090', '17093091', '17093092', '17093093', '17093094', '17093095', '17093096', '17093097', '17093098', '17093099', '17093100', '17093101', '17093102', '17093103', '17093104', '17093105', '17093106', '17093107', '17093108', '17093109', '17093110', '17093113', '17093114', '17093115', '17093116', '17093122', '17093123', '17093124', '17093125', '17093126', '17093127', '17093128', '17093129', '17093130', '17093132', '17093133', '17093135', '17093136', '17093138', '17093140', '17093142', '17093143', '17093145', '17093146', '17093147', '17093117', '17093118', '17093119', '17093120', '17093121', '36385'] proc_lsst.multi INFO: new jobs since last cancel ['17093149', '17093150', '17093151', '17093153', '17093154', '17093155', '17093157', '17093158', '17093159', '17093160', '17093161', '17093162', '17093163', '17093164', '17093165', '17093166', '17093167', '17093168', '17093169', '17093170', '17093171', '17093172', '17093173', '17093174', '17093175', '17093176', '17093177', '17093178', '17093179', '17093180', '17093181', '17093182', '17093183', '17093184', '17093185', '17093186', '17093187', '17093085', '17093086', '17093087', '17093088', '17093089', '17093090', '17093091', '17093092', '17093093', '17093094', '17093095', '17093096', '17093097', '17093098', '17093099', '17093100', '17093101', '17093102', '17093103', '17093104', '17093105', '17093106', '17093107', '17093108', '17093109', '17093110', '17093113', '17093114', '17093115', '17093116', '17093122', '17093123', '17093124', '17093125', '17093126', '17093127', '17093128', '17093129', '17093130', '17093132', '17093133', '17093135', '17093136', '17093138', '17093140', '17093142', '17093143', '17093145', '17093146', '17093147', '17093117', '17093118', '17093119', '17093120', '17093121', '36385'] 2024-03-15 22:49:40 proc_lsst.multi:146 [INFO] found job 17093149 in provider checkpoint proc_lsst.multi INFO: found job 17093149 in provider checkpoint 2024-03-15 22:49:40 proc_lsst.multi:201 [INFO] cancelling 17093149 on provider checkpoint proc_lsst.multi INFO: cancelling 17093149 on provider checkpoint 2024-03-15 22:49:40 proc_lsst.multi:146 [INFO] found job 17093150 in provider checkpoint proc_lsst.multi INFO: found job 17093150 in provider checkpoint 2024-03-15 22:49:40 proc_lsst.multi:201 [INFO] cancelling 17093150 on provider checkpoint proc_lsst.multi INFO: cancelling 17093150 on provider checkpoint 2024-03-15 22:49:40 proc_lsst.multi:146 [INFO] found job 17093151 in provider checkpoint proc_lsst.multi INFO: found job 17093151 in provider checkpoint 2024-03-15 22:49:40 proc_lsst.multi:201 [INFO] cancelling 17093151 on provider checkpoint proc_lsst.multi INFO: cancelling 17093151 on provider checkpoint 2024-03-15 22:49:40 proc_lsst.multi:146 [INFO] found job 17093153 in provider checkpoint proc_lsst.multi INFO: found job 17093153 in provider checkpoint 2024-03-15 22:49:40 proc_lsst.multi:201 [INFO] cancelling 17093153 on provider checkpoint proc_lsst.multi INFO: cancelling 17093153 on provider checkpoint 2024-03-15 22:49:41 proc_lsst.multi:146 [INFO] found job 17093154 in provider checkpoint proc_lsst.multi INFO: found job 17093154 in provider checkpoint 2024-03-15 22:49:41 proc_lsst.multi:201 [INFO] cancelling 17093154 on provider checkpoint proc_lsst.multi INFO: cancelling 17093154 on provider checkpoint 2024-03-15 22:49:41 proc_lsst.multi:146 [INFO] found job 17093155 in provider checkpoint proc_lsst.multi INFO: found job 17093155 in provider checkpoint 2024-03-15 22:49:41 proc_lsst.multi:201 [INFO] cancelling 17093155 on provider checkpoint proc_lsst.multi INFO: cancelling 17093155 on provider checkpoint 2024-03-15 22:49:41 proc_lsst.multi:146 [INFO] found job 17093157 in provider checkpoint proc_lsst.multi INFO: found job 17093157 in provider checkpoint 2024-03-15 22:49:41 proc_lsst.multi:201 [INFO] cancelling 17093157 on provider checkpoint proc_lsst.multi INFO: cancelling 17093157 on provider checkpoint 2024-03-15 22:49:41 proc_lsst.multi:146 [INFO] found job 17093158 in provider checkpoint proc_lsst.multi INFO: found job 17093158 in provider checkpoint 2024-03-15 22:49:41 proc_lsst.multi:201 [INFO] cancelling 17093158 on provider checkpoint proc_lsst.multi INFO: cancelling 17093158 on provider checkpoint 2024-03-15 22:49:41 proc_lsst.multi:146 [INFO] found job 17093159 in provider checkpoint proc_lsst.multi INFO: found job 17093159 in provider checkpoint 2024-03-15 22:49:41 proc_lsst.multi:201 [INFO] cancelling 17093159 on provider checkpoint proc_lsst.multi INFO: cancelling 17093159 on provider checkpoint 2024-03-15 22:49:42 proc_lsst.multi:146 [INFO] found job 17093160 in provider checkpoint proc_lsst.multi INFO: found job 17093160 in provider checkpoint 2024-03-15 22:49:42 proc_lsst.multi:201 [INFO] cancelling 17093160 on provider checkpoint proc_lsst.multi INFO: cancelling 17093160 on provider checkpoint 2024-03-15 22:49:42 proc_lsst.multi:146 [INFO] found job 17093161 in provider checkpoint proc_lsst.multi INFO: found job 17093161 in provider checkpoint 2024-03-15 22:49:42 proc_lsst.multi:201 [INFO] cancelling 17093161 on provider checkpoint proc_lsst.multi INFO: cancelling 17093161 on provider checkpoint 2024-03-15 22:49:42 proc_lsst.multi:146 [INFO] found job 17093162 in provider checkpoint proc_lsst.multi INFO: found job 17093162 in provider checkpoint 2024-03-15 22:49:42 proc_lsst.multi:201 [INFO] cancelling 17093162 on provider checkpoint proc_lsst.multi INFO: cancelling 17093162 on provider checkpoint 2024-03-15 22:49:42 proc_lsst.multi:146 [INFO] found job 17093163 in provider checkpoint proc_lsst.multi INFO: found job 17093163 in provider checkpoint 2024-03-15 22:49:42 proc_lsst.multi:201 [INFO] cancelling 17093163 on provider checkpoint proc_lsst.multi INFO: cancelling 17093163 on provider checkpoint 2024-03-15 22:49:42 proc_lsst.multi:146 [INFO] found job 17093164 in provider checkpoint proc_lsst.multi INFO: found job 17093164 in provider checkpoint 2024-03-15 22:49:42 proc_lsst.multi:201 [INFO] cancelling 17093164 on provider checkpoint proc_lsst.multi INFO: cancelling 17093164 on provider checkpoint 2024-03-15 22:49:43 proc_lsst.multi:146 [INFO] found job 17093165 in provider checkpoint proc_lsst.multi INFO: found job 17093165 in provider checkpoint 2024-03-15 22:49:43 proc_lsst.multi:201 [INFO] cancelling 17093165 on provider checkpoint proc_lsst.multi INFO: cancelling 17093165 on provider checkpoint 2024-03-15 22:49:43 proc_lsst.multi:146 [INFO] found job 17093166 in provider checkpoint proc_lsst.multi INFO: found job 17093166 in provider checkpoint 2024-03-15 22:49:43 proc_lsst.multi:201 [INFO] cancelling 17093166 on provider checkpoint proc_lsst.multi INFO: cancelling 17093166 on provider checkpoint 2024-03-15 22:49:43 proc_lsst.multi:146 [INFO] found job 17093167 in provider checkpoint proc_lsst.multi INFO: found job 17093167 in provider checkpoint 2024-03-15 22:49:43 proc_lsst.multi:201 [INFO] cancelling 17093167 on provider checkpoint proc_lsst.multi INFO: cancelling 17093167 on provider checkpoint 2024-03-15 22:49:43 proc_lsst.multi:146 [INFO] found job 17093168 in provider checkpoint proc_lsst.multi INFO: found job 17093168 in provider checkpoint 2024-03-15 22:49:43 proc_lsst.multi:201 [INFO] cancelling 17093168 on provider checkpoint proc_lsst.multi INFO: cancelling 17093168 on provider checkpoint 2024-03-15 22:49:43 proc_lsst.multi:146 [INFO] found job 17093169 in provider checkpoint proc_lsst.multi INFO: found job 17093169 in provider checkpoint 2024-03-15 22:49:43 proc_lsst.multi:201 [INFO] cancelling 17093169 on provider checkpoint proc_lsst.multi INFO: cancelling 17093169 on provider checkpoint 2024-03-15 22:49:43 proc_lsst.multi:146 [INFO] found job 17093170 in provider checkpoint proc_lsst.multi INFO: found job 17093170 in provider checkpoint 2024-03-15 22:49:43 proc_lsst.multi:201 [INFO] cancelling 17093170 on provider checkpoint proc_lsst.multi INFO: cancelling 17093170 on provider checkpoint 2024-03-15 22:49:44 proc_lsst.multi:146 [INFO] found job 17093171 in provider checkpoint proc_lsst.multi INFO: found job 17093171 in provider checkpoint 2024-03-15 22:49:44 proc_lsst.multi:201 [INFO] cancelling 17093171 on provider checkpoint proc_lsst.multi INFO: cancelling 17093171 on provider checkpoint 2024-03-15 22:49:44 proc_lsst.multi:146 [INFO] found job 17093172 in provider checkpoint proc_lsst.multi INFO: found job 17093172 in provider checkpoint 2024-03-15 22:49:44 proc_lsst.multi:201 [INFO] cancelling 17093172 on provider checkpoint proc_lsst.multi INFO: cancelling 17093172 on provider checkpoint 2024-03-15 22:49:44 proc_lsst.multi:146 [INFO] found job 17093173 in provider checkpoint proc_lsst.multi INFO: found job 17093173 in provider checkpoint 2024-03-15 22:49:44 proc_lsst.multi:201 [INFO] cancelling 17093173 on provider checkpoint proc_lsst.multi INFO: cancelling 17093173 on provider checkpoint 2024-03-15 22:49:44 proc_lsst.multi:146 [INFO] found job 17093174 in provider checkpoint proc_lsst.multi INFO: found job 17093174 in provider checkpoint 2024-03-15 22:49:44 proc_lsst.multi:201 [INFO] cancelling 17093174 on provider checkpoint proc_lsst.multi INFO: cancelling 17093174 on provider checkpoint 2024-03-15 22:49:44 proc_lsst.multi:146 [INFO] found job 17093175 in provider checkpoint proc_lsst.multi INFO: found job 17093175 in provider checkpoint 2024-03-15 22:49:44 proc_lsst.multi:201 [INFO] cancelling 17093175 on provider checkpoint proc_lsst.multi INFO: cancelling 17093175 on provider checkpoint 2024-03-15 22:49:45 proc_lsst.multi:146 [INFO] found job 17093176 in provider checkpoint proc_lsst.multi INFO: found job 17093176 in provider checkpoint 2024-03-15 22:49:45 proc_lsst.multi:201 [INFO] cancelling 17093176 on provider checkpoint proc_lsst.multi INFO: cancelling 17093176 on provider checkpoint 2024-03-15 22:49:45 proc_lsst.multi:146 [INFO] found job 17093177 in provider checkpoint proc_lsst.multi INFO: found job 17093177 in provider checkpoint 2024-03-15 22:49:45 proc_lsst.multi:201 [INFO] cancelling 17093177 on provider checkpoint proc_lsst.multi INFO: cancelling 17093177 on provider checkpoint 2024-03-15 22:49:45 proc_lsst.multi:146 [INFO] found job 17093178 in provider checkpoint proc_lsst.multi INFO: found job 17093178 in provider checkpoint 2024-03-15 22:49:45 proc_lsst.multi:201 [INFO] cancelling 17093178 on provider checkpoint proc_lsst.multi INFO: cancelling 17093178 on provider checkpoint 2024-03-15 22:49:45 proc_lsst.multi:146 [INFO] found job 17093179 in provider checkpoint proc_lsst.multi INFO: found job 17093179 in provider checkpoint 2024-03-15 22:49:45 proc_lsst.multi:201 [INFO] cancelling 17093179 on provider checkpoint proc_lsst.multi INFO: cancelling 17093179 on provider checkpoint 2024-03-15 22:49:45 proc_lsst.multi:146 [INFO] found job 17093180 in provider checkpoint proc_lsst.multi INFO: found job 17093180 in provider checkpoint 2024-03-15 22:49:45 proc_lsst.multi:201 [INFO] cancelling 17093180 on provider checkpoint proc_lsst.multi INFO: cancelling 17093180 on provider checkpoint 2024-03-15 22:49:45 proc_lsst.multi:146 [INFO] found job 17093181 in provider checkpoint proc_lsst.multi INFO: found job 17093181 in provider checkpoint 2024-03-15 22:49:45 proc_lsst.multi:201 [INFO] cancelling 17093181 on provider checkpoint proc_lsst.multi INFO: cancelling 17093181 on provider checkpoint 2024-03-15 22:49:46 proc_lsst.multi:146 [INFO] found job 17093182 in provider checkpoint proc_lsst.multi INFO: found job 17093182 in provider checkpoint 2024-03-15 22:49:46 proc_lsst.multi:201 [INFO] cancelling 17093182 on provider checkpoint proc_lsst.multi INFO: cancelling 17093182 on provider checkpoint 2024-03-15 22:49:46 proc_lsst.multi:146 [INFO] found job 17093183 in provider checkpoint proc_lsst.multi INFO: found job 17093183 in provider checkpoint 2024-03-15 22:49:46 proc_lsst.multi:201 [INFO] cancelling 17093183 on provider checkpoint proc_lsst.multi INFO: cancelling 17093183 on provider checkpoint 2024-03-15 22:49:46 proc_lsst.multi:146 [INFO] found job 17093184 in provider checkpoint proc_lsst.multi INFO: found job 17093184 in provider checkpoint 2024-03-15 22:49:46 proc_lsst.multi:201 [INFO] cancelling 17093184 on provider checkpoint proc_lsst.multi INFO: cancelling 17093184 on provider checkpoint 2024-03-15 22:49:46 proc_lsst.multi:146 [INFO] found job 17093185 in provider checkpoint proc_lsst.multi INFO: found job 17093185 in provider checkpoint 2024-03-15 22:49:46 proc_lsst.multi:201 [INFO] cancelling 17093185 on provider checkpoint proc_lsst.multi INFO: cancelling 17093185 on provider checkpoint 2024-03-15 22:49:46 proc_lsst.multi:146 [INFO] found job 17093186 in provider checkpoint proc_lsst.multi INFO: found job 17093186 in provider checkpoint 2024-03-15 22:49:46 proc_lsst.multi:201 [INFO] cancelling 17093186 on provider checkpoint proc_lsst.multi INFO: cancelling 17093186 on provider checkpoint 2024-03-15 22:49:46 proc_lsst.multi:146 [INFO] found job 17093187 in provider checkpoint proc_lsst.multi INFO: found job 17093187 in provider checkpoint 2024-03-15 22:49:46 proc_lsst.multi:201 [INFO] cancelling 17093187 on provider checkpoint proc_lsst.multi INFO: cancelling 17093187 on provider checkpoint 2024-03-15 22:49:47 proc_lsst.multi:146 [INFO] found job 17093085 in provider astro proc_lsst.multi INFO: found job 17093085 in provider astro 2024-03-15 22:49:47 proc_lsst.multi:201 [INFO] cancelling 17093085 on provider astro proc_lsst.multi INFO: cancelling 17093085 on provider astro 2024-03-15 22:49:47 proc_lsst.multi:146 [INFO] found job 17093086 in provider astro proc_lsst.multi INFO: found job 17093086 in provider astro 2024-03-15 22:49:47 proc_lsst.multi:201 [INFO] cancelling 17093086 on provider astro proc_lsst.multi INFO: cancelling 17093086 on provider astro 2024-03-15 22:49:47 proc_lsst.multi:146 [INFO] found job 17093087 in provider astro proc_lsst.multi INFO: found job 17093087 in provider astro 2024-03-15 22:49:47 proc_lsst.multi:201 [INFO] cancelling 17093087 on provider astro proc_lsst.multi INFO: cancelling 17093087 on provider astro 2024-03-15 22:49:47 proc_lsst.multi:146 [INFO] found job 17093088 in provider astro proc_lsst.multi INFO: found job 17093088 in provider astro 2024-03-15 22:49:47 proc_lsst.multi:201 [INFO] cancelling 17093088 on provider astro proc_lsst.multi INFO: cancelling 17093088 on provider astro 2024-03-15 22:49:47 proc_lsst.multi:146 [INFO] found job 17093089 in provider astro proc_lsst.multi INFO: found job 17093089 in provider astro 2024-03-15 22:49:47 proc_lsst.multi:201 [INFO] cancelling 17093089 on provider astro proc_lsst.multi INFO: cancelling 17093089 on provider astro 2024-03-15 22:49:47 proc_lsst.multi:146 [INFO] found job 17093090 in provider astro proc_lsst.multi INFO: found job 17093090 in provider astro 2024-03-15 22:49:47 proc_lsst.multi:201 [INFO] cancelling 17093090 on provider astro proc_lsst.multi INFO: cancelling 17093090 on provider astro 2024-03-15 22:49:48 proc_lsst.multi:146 [INFO] found job 17093091 in provider astro proc_lsst.multi INFO: found job 17093091 in provider astro 2024-03-15 22:49:48 proc_lsst.multi:201 [INFO] cancelling 17093091 on provider astro proc_lsst.multi INFO: cancelling 17093091 on provider astro 2024-03-15 22:49:48 proc_lsst.multi:146 [INFO] found job 17093092 in provider astro proc_lsst.multi INFO: found job 17093092 in provider astro 2024-03-15 22:49:48 proc_lsst.multi:201 [INFO] cancelling 17093092 on provider astro proc_lsst.multi INFO: cancelling 17093092 on provider astro 2024-03-15 22:49:48 proc_lsst.multi:146 [INFO] found job 17093093 in provider astro proc_lsst.multi INFO: found job 17093093 in provider astro 2024-03-15 22:49:48 proc_lsst.multi:201 [INFO] cancelling 17093093 on provider astro proc_lsst.multi INFO: cancelling 17093093 on provider astro 2024-03-15 22:49:48 proc_lsst.multi:146 [INFO] found job 17093094 in provider astro proc_lsst.multi INFO: found job 17093094 in provider astro 2024-03-15 22:49:48 proc_lsst.multi:201 [INFO] cancelling 17093094 on provider astro proc_lsst.multi INFO: cancelling 17093094 on provider astro 2024-03-15 22:49:48 proc_lsst.multi:146 [INFO] found job 17093095 in provider astro proc_lsst.multi INFO: found job 17093095 in provider astro 2024-03-15 22:49:48 proc_lsst.multi:201 [INFO] cancelling 17093095 on provider astro proc_lsst.multi INFO: cancelling 17093095 on provider astro 2024-03-15 22:49:48 proc_lsst.multi:146 [INFO] found job 17093096 in provider astro proc_lsst.multi INFO: found job 17093096 in provider astro 2024-03-15 22:49:48 proc_lsst.multi:201 [INFO] cancelling 17093096 on provider astro proc_lsst.multi INFO: cancelling 17093096 on provider astro 2024-03-15 22:49:49 proc_lsst.multi:146 [INFO] found job 17093097 in provider astro proc_lsst.multi INFO: found job 17093097 in provider astro 2024-03-15 22:49:49 proc_lsst.multi:201 [INFO] cancelling 17093097 on provider astro proc_lsst.multi INFO: cancelling 17093097 on provider astro 2024-03-15 22:49:49 proc_lsst.multi:146 [INFO] found job 17093098 in provider astro proc_lsst.multi INFO: found job 17093098 in provider astro 2024-03-15 22:49:49 proc_lsst.multi:201 [INFO] cancelling 17093098 on provider astro proc_lsst.multi INFO: cancelling 17093098 on provider astro 2024-03-15 22:49:49 proc_lsst.multi:146 [INFO] found job 17093099 in provider astro proc_lsst.multi INFO: found job 17093099 in provider astro 2024-03-15 22:49:49 proc_lsst.multi:201 [INFO] cancelling 17093099 on provider astro proc_lsst.multi INFO: cancelling 17093099 on provider astro 2024-03-15 22:49:49 proc_lsst.multi:146 [INFO] found job 17093100 in provider astro proc_lsst.multi INFO: found job 17093100 in provider astro 2024-03-15 22:49:49 proc_lsst.multi:201 [INFO] cancelling 17093100 on provider astro proc_lsst.multi INFO: cancelling 17093100 on provider astro 2024-03-15 22:49:49 proc_lsst.multi:146 [INFO] found job 17093101 in provider astro proc_lsst.multi INFO: found job 17093101 in provider astro 2024-03-15 22:49:49 proc_lsst.multi:201 [INFO] cancelling 17093101 on provider astro proc_lsst.multi INFO: cancelling 17093101 on provider astro 2024-03-15 22:49:49 proc_lsst.multi:146 [INFO] found job 17093102 in provider astro proc_lsst.multi INFO: found job 17093102 in provider astro 2024-03-15 22:49:49 proc_lsst.multi:201 [INFO] cancelling 17093102 on provider astro proc_lsst.multi INFO: cancelling 17093102 on provider astro 2024-03-15 22:49:50 proc_lsst.multi:146 [INFO] found job 17093103 in provider astro proc_lsst.multi INFO: found job 17093103 in provider astro 2024-03-15 22:49:50 proc_lsst.multi:201 [INFO] cancelling 17093103 on provider astro proc_lsst.multi INFO: cancelling 17093103 on provider astro 2024-03-15 22:49:50 proc_lsst.multi:146 [INFO] found job 17093104 in provider astro proc_lsst.multi INFO: found job 17093104 in provider astro 2024-03-15 22:49:50 proc_lsst.multi:201 [INFO] cancelling 17093104 on provider astro proc_lsst.multi INFO: cancelling 17093104 on provider astro 2024-03-15 22:49:50 proc_lsst.multi:146 [INFO] found job 17093105 in provider astro proc_lsst.multi INFO: found job 17093105 in provider astro 2024-03-15 22:49:50 proc_lsst.multi:201 [INFO] cancelling 17093105 on provider astro proc_lsst.multi INFO: cancelling 17093105 on provider astro 2024-03-15 22:49:50 proc_lsst.multi:146 [INFO] found job 17093106 in provider astro proc_lsst.multi INFO: found job 17093106 in provider astro 2024-03-15 22:49:50 proc_lsst.multi:201 [INFO] cancelling 17093106 on provider astro proc_lsst.multi INFO: cancelling 17093106 on provider astro 2024-03-15 22:49:50 proc_lsst.multi:146 [INFO] found job 17093107 in provider astro proc_lsst.multi INFO: found job 17093107 in provider astro 2024-03-15 22:49:50 proc_lsst.multi:201 [INFO] cancelling 17093107 on provider astro proc_lsst.multi INFO: cancelling 17093107 on provider astro 2024-03-15 22:49:50 proc_lsst.multi:146 [INFO] found job 17093108 in provider astro proc_lsst.multi INFO: found job 17093108 in provider astro 2024-03-15 22:49:50 proc_lsst.multi:201 [INFO] cancelling 17093108 on provider astro proc_lsst.multi INFO: cancelling 17093108 on provider astro 2024-03-15 22:49:51 proc_lsst.multi:146 [INFO] found job 17093109 in provider astro proc_lsst.multi INFO: found job 17093109 in provider astro 2024-03-15 22:49:51 proc_lsst.multi:201 [INFO] cancelling 17093109 on provider astro proc_lsst.multi INFO: cancelling 17093109 on provider astro 2024-03-15 22:49:51 proc_lsst.multi:146 [INFO] found job 17093110 in provider astro proc_lsst.multi INFO: found job 17093110 in provider astro 2024-03-15 22:49:51 proc_lsst.multi:201 [INFO] cancelling 17093110 on provider astro proc_lsst.multi INFO: cancelling 17093110 on provider astro 2024-03-15 22:49:51 proc_lsst.multi:146 [INFO] found job 17093113 in provider astro proc_lsst.multi INFO: found job 17093113 in provider astro 2024-03-15 22:49:51 proc_lsst.multi:201 [INFO] cancelling 17093113 on provider astro proc_lsst.multi INFO: cancelling 17093113 on provider astro 2024-03-15 22:49:51 proc_lsst.multi:146 [INFO] found job 17093114 in provider astro proc_lsst.multi INFO: found job 17093114 in provider astro 2024-03-15 22:49:51 proc_lsst.multi:201 [INFO] cancelling 17093114 on provider astro proc_lsst.multi INFO: cancelling 17093114 on provider astro 2024-03-15 22:49:51 proc_lsst.multi:146 [INFO] found job 17093115 in provider astro proc_lsst.multi INFO: found job 17093115 in provider astro 2024-03-15 22:49:51 proc_lsst.multi:201 [INFO] cancelling 17093115 on provider astro proc_lsst.multi INFO: cancelling 17093115 on provider astro 2024-03-15 22:49:51 proc_lsst.multi:146 [INFO] found job 17093116 in provider astro proc_lsst.multi INFO: found job 17093116 in provider astro 2024-03-15 22:49:51 proc_lsst.multi:201 [INFO] cancelling 17093116 on provider astro proc_lsst.multi INFO: cancelling 17093116 on provider astro 2024-03-15 22:49:52 proc_lsst.multi:146 [INFO] found job 17093122 in provider escience_a40 proc_lsst.multi INFO: found job 17093122 in provider escience_a40 2024-03-15 22:49:52 proc_lsst.multi:201 [INFO] cancelling 17093122 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093122 on provider escience_a40 2024-03-15 22:49:52 proc_lsst.multi:146 [INFO] found job 17093123 in provider escience_a40 proc_lsst.multi INFO: found job 17093123 in provider escience_a40 2024-03-15 22:49:52 proc_lsst.multi:201 [INFO] cancelling 17093123 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093123 on provider escience_a40 2024-03-15 22:49:52 proc_lsst.multi:146 [INFO] found job 17093124 in provider escience_a40 proc_lsst.multi INFO: found job 17093124 in provider escience_a40 2024-03-15 22:49:52 proc_lsst.multi:201 [INFO] cancelling 17093124 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093124 on provider escience_a40 2024-03-15 22:49:52 proc_lsst.multi:146 [INFO] found job 17093125 in provider escience_a40 proc_lsst.multi INFO: found job 17093125 in provider escience_a40 2024-03-15 22:49:52 proc_lsst.multi:201 [INFO] cancelling 17093125 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093125 on provider escience_a40 2024-03-15 22:49:52 proc_lsst.multi:146 [INFO] found job 17093126 in provider escience_a40 proc_lsst.multi INFO: found job 17093126 in provider escience_a40 2024-03-15 22:49:52 proc_lsst.multi:201 [INFO] cancelling 17093126 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093126 on provider escience_a40 2024-03-15 22:49:52 proc_lsst.multi:146 [INFO] found job 17093127 in provider escience_a40 proc_lsst.multi INFO: found job 17093127 in provider escience_a40 2024-03-15 22:49:52 proc_lsst.multi:201 [INFO] cancelling 17093127 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093127 on provider escience_a40 2024-03-15 22:49:53 proc_lsst.multi:146 [INFO] found job 17093128 in provider escience_a40 proc_lsst.multi INFO: found job 17093128 in provider escience_a40 2024-03-15 22:49:53 proc_lsst.multi:201 [INFO] cancelling 17093128 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093128 on provider escience_a40 2024-03-15 22:49:53 proc_lsst.multi:146 [INFO] found job 17093129 in provider escience_a40 proc_lsst.multi INFO: found job 17093129 in provider escience_a40 2024-03-15 22:49:53 proc_lsst.multi:201 [INFO] cancelling 17093129 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093129 on provider escience_a40 2024-03-15 22:49:53 proc_lsst.multi:146 [INFO] found job 17093130 in provider escience_a40 proc_lsst.multi INFO: found job 17093130 in provider escience_a40 2024-03-15 22:49:53 proc_lsst.multi:201 [INFO] cancelling 17093130 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093130 on provider escience_a40 2024-03-15 22:49:53 proc_lsst.multi:146 [INFO] found job 17093132 in provider escience_a40 proc_lsst.multi INFO: found job 17093132 in provider escience_a40 2024-03-15 22:49:53 proc_lsst.multi:201 [INFO] cancelling 17093132 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093132 on provider escience_a40 2024-03-15 22:49:53 proc_lsst.multi:146 [INFO] found job 17093133 in provider escience_a40 proc_lsst.multi INFO: found job 17093133 in provider escience_a40 2024-03-15 22:49:53 proc_lsst.multi:201 [INFO] cancelling 17093133 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093133 on provider escience_a40 2024-03-15 22:49:53 proc_lsst.multi:146 [INFO] found job 17093135 in provider escience_a40 proc_lsst.multi INFO: found job 17093135 in provider escience_a40 2024-03-15 22:49:53 proc_lsst.multi:201 [INFO] cancelling 17093135 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093135 on provider escience_a40 2024-03-15 22:49:54 proc_lsst.multi:146 [INFO] found job 17093136 in provider escience_a40 proc_lsst.multi INFO: found job 17093136 in provider escience_a40 2024-03-15 22:49:54 proc_lsst.multi:201 [INFO] cancelling 17093136 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093136 on provider escience_a40 2024-03-15 22:49:54 proc_lsst.multi:146 [INFO] found job 17093138 in provider escience_a40 proc_lsst.multi INFO: found job 17093138 in provider escience_a40 2024-03-15 22:49:54 proc_lsst.multi:201 [INFO] cancelling 17093138 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093138 on provider escience_a40 2024-03-15 22:49:54 proc_lsst.multi:146 [INFO] found job 17093140 in provider escience_a40 proc_lsst.multi INFO: found job 17093140 in provider escience_a40 2024-03-15 22:49:54 proc_lsst.multi:201 [INFO] cancelling 17093140 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093140 on provider escience_a40 2024-03-15 22:49:54 proc_lsst.multi:146 [INFO] found job 17093142 in provider escience_a40 proc_lsst.multi INFO: found job 17093142 in provider escience_a40 2024-03-15 22:49:54 proc_lsst.multi:201 [INFO] cancelling 17093142 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093142 on provider escience_a40 2024-03-15 22:49:54 proc_lsst.multi:146 [INFO] found job 17093143 in provider escience_a40 proc_lsst.multi INFO: found job 17093143 in provider escience_a40 2024-03-15 22:49:54 proc_lsst.multi:201 [INFO] cancelling 17093143 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093143 on provider escience_a40 2024-03-15 22:49:55 proc_lsst.multi:146 [INFO] found job 17093145 in provider escience_a40 proc_lsst.multi INFO: found job 17093145 in provider escience_a40 2024-03-15 22:49:55 proc_lsst.multi:201 [INFO] cancelling 17093145 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093145 on provider escience_a40 2024-03-15 22:49:55 proc_lsst.multi:146 [INFO] found job 17093146 in provider escience_a40 proc_lsst.multi INFO: found job 17093146 in provider escience_a40 2024-03-15 22:49:55 proc_lsst.multi:201 [INFO] cancelling 17093146 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093146 on provider escience_a40 2024-03-15 22:49:55 proc_lsst.multi:146 [INFO] found job 17093147 in provider escience_a40 proc_lsst.multi INFO: found job 17093147 in provider escience_a40 2024-03-15 22:49:55 proc_lsst.multi:201 [INFO] cancelling 17093147 on provider escience_a40 proc_lsst.multi INFO: cancelling 17093147 on provider escience_a40 2024-03-15 22:49:55 proc_lsst.multi:146 [INFO] found job 17093117 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093117 in provider escience_rtx6k 2024-03-15 22:49:55 proc_lsst.multi:201 [INFO] cancelling 17093117 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17093117 on provider escience_rtx6k 2024-03-15 22:49:55 proc_lsst.multi:146 [INFO] found job 17093118 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093118 in provider escience_rtx6k 2024-03-15 22:49:55 proc_lsst.multi:201 [INFO] cancelling 17093118 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17093118 on provider escience_rtx6k 2024-03-15 22:49:55 proc_lsst.multi:146 [INFO] found job 17093119 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093119 in provider escience_rtx6k 2024-03-15 22:49:55 proc_lsst.multi:201 [INFO] cancelling 17093119 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17093119 on provider escience_rtx6k 2024-03-15 22:49:56 proc_lsst.multi:146 [INFO] found job 17093120 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093120 in provider escience_rtx6k 2024-03-15 22:49:56 proc_lsst.multi:201 [INFO] cancelling 17093120 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17093120 on provider escience_rtx6k 2024-03-15 22:49:56 proc_lsst.multi:146 [INFO] found job 17093121 in provider escience_rtx6k proc_lsst.multi INFO: found job 17093121 in provider escience_rtx6k 2024-03-15 22:49:56 proc_lsst.multi:201 [INFO] cancelling 17093121 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17093121 on provider escience_rtx6k 2024-03-15 22:49:56 proc_lsst.multi:146 [INFO] found job 36385 in provider local proc_lsst.multi INFO: found job 36385 in provider local 2024-03-15 22:49:56 proc_lsst.multi:201 [INFO] cancelling 36385 on provider local proc_lsst.multi INFO: cancelling 36385 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 36385 parsl.providers.local.local WARNING: Failed to kill PID: 36385 and child processes on local 2024-03-15 22:49:57 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 192.6109 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20210516_A0g_science#step0_20240315T224631Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 192.6195 seconds; current memory usage: 0.275 Gibyte, delta: 0.001 Gibyte, peak delta: 0.003 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 204.6823 seconds; current memory usage: 0.275 Gibyte, delta: 0.138 Gibyte, peak delta: 0.140 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.277 Gibyte (main), 0.277 Gibyte (largest child process) Run Id: None Run Name: DEEP_20210516_A0g_science#step0_20240315T224631Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 3m27.005s user 0m56.938s sys 0m29.524s