+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20220527/A0b --output-run DEEP/20220527/A0b/science#step0/20240319T003842Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 14.9653 seconds; current memory usage: 0.178 Gibyte, delta: 0.041 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0036 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0095 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/pipeline_orig.qgraph DEEP/20220527/A0b/science#step0/20240319T003842Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 2.5610 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 6.7521 seconds; current memory usage: 0.272 Gibyte, delta: 0.095 Gibyte, peak delta: 0.082 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.263 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0104 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.263 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 248 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20220527_A0b_science#step0_20240319T003842Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.0580 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.263 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 250 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.6448 seconds; current memory usage: 0.274 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.274 Gibyte (main), 0.263 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 258633a7-fe3f-4932-be16-2c39f24b8ecb parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-19 00:39:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-19 00:39:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-19 00:39:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-19 00:39:23 proc_lsst.multi:170 [INFO] [multi] job_id 96275 proc_lsst.multi INFO: [multi] job_id 96275 2024-03-19 00:39:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:23 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 96275 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 96275 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 96275 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 187b2207b41f584570e9524a7126ccca parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b42980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/f34031eb-b745-4710-aace-fa4857ebaf3e_isrForCrosstalkSources_1101474_62.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/f34031eb-b745-4710-aace-fa4857ebaf3e_isrForCrosstalkSources_1101474_62.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 401f46446f8e1eb11d5c6cf8e53e52f9 parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b42ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/a2c71ca4-7f0a-486f-9623-512ca6ff9e5c_isrForCrosstalkSources_1101468_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/a2c71ca4-7f0a-486f-9623-512ca6ff9e5c_isrForCrosstalkSources_1101468_28.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash c57ece3be8807998ad6b2638121e7ffc parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b42fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/7690426e-d29a-42fd-891d-7562a610f3f7_isrForCrosstalkSources_1101465_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/7690426e-d29a-42fd-891d-7562a610f3f7_isrForCrosstalkSources_1101465_33.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash bec4a4600289c4a46095f96d59c8d8b5 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b42e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/9bc00ad6-7e62-496b-920c-fd7a79299682_isrForCrosstalkSources_1101471_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/9bc00ad6-7e62-496b-920c-fd7a79299682_isrForCrosstalkSources_1101471_15.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash ddc4c128e66255f8445febcb96a8dc7f parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b42f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/ea33079e-7f82-4c06-b8e8-e2bf6911d196_isrForCrosstalkSources_1101471_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/ea33079e-7f82-4c06-b8e8-e2bf6911d196_isrForCrosstalkSources_1101471_45.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 3d65d317806d8bed9a87720d75e1d2fa parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b43100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/3fcdb653-8fec-4418-b3a4-07fe78650f30_isrForCrosstalkSources_1101465_21.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/3fcdb653-8fec-4418-b3a4-07fe78650f30_isrForCrosstalkSources_1101465_21.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 1bd3b1844a244ad8246083d41746d753 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b431a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/c6ebf8d4-fdae-4abb-95a7-437a0bf19c00_isrForCrosstalkSources_1101468_51.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/c6ebf8d4-fdae-4abb-95a7-437a0bf19c00_isrForCrosstalkSources_1101468_51.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 7fe5b352644fc8369d81d53a28a5b25a parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b422a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/079368a0-64dc-4819-a6ce-804316ecfb41_isrForCrosstalkSources_1101474_20.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/079368a0-64dc-4819-a6ce-804316ecfb41_isrForCrosstalkSources_1101474_20.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 4ea7bae8134bed0da84db7eb7cc963d3 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b42b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/8f3e22ea-1058-4ed8-9958-7fc76886df29_isrForCrosstalkSources_1101474_45.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/8f3e22ea-1058-4ed8-9958-7fc76886df29_isrForCrosstalkSources_1101474_45.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash bd0cf8a26f1583b5cedfc89c9e4ac9bd parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b432e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/3d573f99-e69e-4188-8510-bbc807af21c0_isrForCrosstalkSources_1101468_8.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/3d573f99-e69e-4188-8510-bbc807af21c0_isrForCrosstalkSources_1101468_8.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash c9cbaef72fd3f7334579355d731122d2 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b43060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/ada1bee2-b2ad-45a7-a7ac-81846d5d20ac_isrForCrosstalkSources_1101474_26.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/ada1bee2-b2ad-45a7-a7ac-81846d5d20ac_isrForCrosstalkSources_1101474_26.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 1bf8984b1a802f881b07398e5855452d parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b43420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/75ab27aa-888e-402a-b904-c90c4bc6b0a4_isrForCrosstalkSources_1101471_58.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/75ab27aa-888e-402a-b904-c90c4bc6b0a4_isrForCrosstalkSources_1101471_58.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 34caeaec523ab66c2754231aacbd82c0 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b43600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/b65b1bbd-17da-40da-8f08-68eec6af7717_isrForCrosstalkSources_1101474_56.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/b65b1bbd-17da-40da-8f08-68eec6af7717_isrForCrosstalkSources_1101474_56.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash af874b35edd16f63b836b17c4fbda0d0 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b437e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/1dad9f89-3f4c-420b-83cd-c61d13cef7ba_isrForCrosstalkSources_1101465_1.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/1dad9f89-3f4c-420b-83cd-c61d13cef7ba_isrForCrosstalkSources_1101465_1.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 01fc1dc6a02b5ecea9ab0615d3b16520 parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b439c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/90db54a6-590f-47e2-9eaf-5d1db4da62e7_isrForCrosstalkSources_1101465_30.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/90db54a6-590f-47e2-9eaf-5d1db4da62e7_isrForCrosstalkSources_1101465_30.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 2ad61a18f290e2ee1d301668523c89f6 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b43560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/0443233d-e633-419d-bfb4-f39eadc00931_isrForCrosstalkSources_1101465_41.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/0443233d-e633-419d-bfb4-f39eadc00931_isrForCrosstalkSources_1101465_41.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 4579096a32c9f2155abc0898b87b8989 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b43a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/8a2a4216-09ec-4a0d-b6f2-6f6ebafc8d97_isrForCrosstalkSources_1101471_17.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/8a2a4216-09ec-4a0d-b6f2-6f6ebafc8d97_isrForCrosstalkSources_1101471_17.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash ecc2f3cd11a6058bc67602d019b5094e parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b43b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/58c43dfa-25ba-4e98-bfaf-d602e12b734d_isrForCrosstalkSources_1101468_53.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/58c43dfa-25ba-4e98-bfaf-d602e12b734d_isrForCrosstalkSources_1101468_53.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 3645338f1d9360d733051d99769d99b8 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b43c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/08d80e02-45cf-4cca-8249-7781459129d4_isrForCrosstalkSources_1101471_14.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/08d80e02-45cf-4cca-8249-7781459129d4_isrForCrosstalkSources_1101471_14.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 2e01c129e88b09351e325eba20d8da69 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b43920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/91f19c82-79bf-4a9f-b49f-444f5f4d4309_isrForCrosstalkSources_1101465_39.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/91f19c82-79bf-4a9f-b49f-444f5f4d4309_isrForCrosstalkSources_1101465_39.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 03e7f0287476af7ab5bce4cc088f8681 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b43d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/53d19e00-ed9a-41b2-a2ef-19ac81346082_isrForCrosstalkSources_1101465_51.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/53d19e00-ed9a-41b2-a2ef-19ac81346082_isrForCrosstalkSources_1101465_51.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash dd5b65db4f79925c223ae32e40a498c4 parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b43ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/3fd32e61-2876-4daf-9243-880b8f11f6a3_isrForCrosstalkSources_1101465_22.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/3fd32e61-2876-4daf-9243-880b8f11f6a3_isrForCrosstalkSources_1101465_22.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash c8e49dbe01320cd5d820fc964fdea43c parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b43e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/f59944fb-c5e2-46ee-9570-41183756ec81_isrForCrosstalkSources_1101468_58.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/f59944fb-c5e2-46ee-9570-41183756ec81_isrForCrosstalkSources_1101468_58.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 6bb86d794670500444c2293f051d7773 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b43ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/5b04112e-7777-49c3-a791-fad4b0097801_isrForCrosstalkSources_1101474_24.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/5b04112e-7777-49c3-a791-fad4b0097801_isrForCrosstalkSources_1101474_24.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash df00fcc848cc2da0be585a90daa25ba3 parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498b43ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/da9a2325-ff7f-4a71-a9df-dd5b766ee272_isrForCrosstalkSources_1101468_21.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/da9a2325-ff7f-4a71-a9df-dd5b766ee272_isrForCrosstalkSources_1101468_21.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 1c8943d959ad15bf803c7a4b10d32978 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/96f47baa-aa21-4e03-ad9c-2c09beea2ff4_isrForCrosstalkSources_1101471_48.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/96f47baa-aa21-4e03-ad9c-2c09beea2ff4_isrForCrosstalkSources_1101471_48.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 7a09b168ce6a073b98ea89be92acde44 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/21e7102d-1330-4485-9db0-a7e7c9775603_isrForCrosstalkSources_1101474_38.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/21e7102d-1330-4485-9db0-a7e7c9775603_isrForCrosstalkSources_1101474_38.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash d269a0f5cf38f0e4a32218a78061a074 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/2514b612-4df2-4884-bbde-0a264749760b_isrForCrosstalkSources_1101471_9.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/2514b612-4df2-4884-bbde-0a264749760b_isrForCrosstalkSources_1101471_9.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 08c848e45a8c9abfc2d81e2e06999488 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/452ba558-1c65-4489-a013-10f154a695e4_isrForCrosstalkSources_1101465_58.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/452ba558-1c65-4489-a013-10f154a695e4_isrForCrosstalkSources_1101465_58.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 22ef6819bc6b40a866cda31d63719ae2 parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/f65cae74-693b-4883-9f31-d3d127210deb_isrForCrosstalkSources_1101471_7.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/f65cae74-693b-4883-9f31-d3d127210deb_isrForCrosstalkSources_1101471_7.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash e15c86ecfa7825893ca5bf4261d2f03b parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/4fad8c51-51c3-47c0-9d36-64a650ddc82d_isrForCrosstalkSources_1101474_58.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/4fad8c51-51c3-47c0-9d36-64a650ddc82d_isrForCrosstalkSources_1101474_58.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 50edf6b5cabbb72c4c82116af55d8895 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/03ef263c-ec1a-45ad-b3ef-b62b116c58da_isrForCrosstalkSources_1101474_9.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/03ef263c-ec1a-45ad-b3ef-b62b116c58da_isrForCrosstalkSources_1101474_9.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash e4d54e80703572fe8ac4b857741a5a02 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/481db7a6-f89b-4f7c-8d1b-b035996c2253_isrForCrosstalkSources_1101474_32.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/481db7a6-f89b-4f7c-8d1b-b035996c2253_isrForCrosstalkSources_1101474_32.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 9752593830886e12f8af795df6febbd8 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/b747d873-d0c7-4ee0-b41f-f06ec1aa2919_isrForCrosstalkSources_1101471_31.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/b747d873-d0c7-4ee0-b41f-f06ec1aa2919_isrForCrosstalkSources_1101471_31.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 362f67f9ad95c3afb163ae96bcead803 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/4c330de1-9f83-467e-a472-3003f9d17445_isrForCrosstalkSources_1101468_22.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/4c330de1-9f83-467e-a472-3003f9d17445_isrForCrosstalkSources_1101468_22.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash 0f772136bd2e20fa8b9d61b67bb27489 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/dac31026-f676-4ec3-96b4-850dcc2bb1fd_isrForCrosstalkSources_1101468_55.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/dac31026-f676-4ec3-96b4-850dcc2bb1fd_isrForCrosstalkSources_1101468_55.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash b926f259150611a59dabe97d779adf63 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/d4f919d9-12df-432d-bd25-f8a9f587ccb1_isrForCrosstalkSources_1101465_38.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/d4f919d9-12df-432d-bd25-f8a9f587ccb1_isrForCrosstalkSources_1101465_38.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 2df286b16a69736692cdfc315b2c0557 parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/2b3f00f9-9e0e-4f8d-a2a3-2600c7d869e0_isrForCrosstalkSources_1101471_38.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/2b3f00f9-9e0e-4f8d-a2a3-2600c7d869e0_isrForCrosstalkSources_1101471_38.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 50686e686d1cfb4ad9db8ae3dc5934b5 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/f7af661a-826b-465d-ba3a-e4d38b008f33_isrForCrosstalkSources_1101471_41.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/f7af661a-826b-465d-ba3a-e4d38b008f33_isrForCrosstalkSources_1101471_41.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash a0c3634aa3a5d09105198068f56c0c97 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/b03b4d62-dfe4-4ca6-93b6-c6c543b9bd06_isrForCrosstalkSources_1101468_57.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/b03b4d62-dfe4-4ca6-93b6-c6c543b9bd06_isrForCrosstalkSources_1101468_57.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash fbca22301eb17297b9c69645b72fe4d6 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/a7eb4196-3a5d-4268-8296-4182c1bd34f9_isrForCrosstalkSources_1101465_29.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/a7eb4196-3a5d-4268-8296-4182c1bd34f9_isrForCrosstalkSources_1101465_29.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 1a20821b5cc75a5bd4efc73c3d088b1a parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/fae3e0f8-3ded-49d4-987e-7b8cdcbec387_isrForCrosstalkSources_1101465_16.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/fae3e0f8-3ded-49d4-987e-7b8cdcbec387_isrForCrosstalkSources_1101465_16.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 99a9b0ddd7a7db5a222069a1560b1051 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/a96d5af6-87a1-49d1-bd8a-43b30042b9b2_isrForCrosstalkSources_1101468_62.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/a96d5af6-87a1-49d1-bd8a-43b30042b9b2_isrForCrosstalkSources_1101468_62.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 341b5a088027e676543e82a796c85832 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/d2c7da22-3f60-4d1e-a727-607eff679943_isrForCrosstalkSources_1101471_44.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/d2c7da22-3f60-4d1e-a727-607eff679943_isrForCrosstalkSources_1101471_44.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 8b86a8742115c30cdc6ee98b28001d7f parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/800eb2de-e042-4ed3-8ccb-14287b43a7b1_isrForCrosstalkSources_1101465_7.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/800eb2de-e042-4ed3-8ccb-14287b43a7b1_isrForCrosstalkSources_1101465_7.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash bef2054d013be25abf03fbd4ee0abd2f parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/a8ee3cec-cdaa-4cb2-846d-0f9643cc54af_isrForCrosstalkSources_1101465_43.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/a8ee3cec-cdaa-4cb2-846d-0f9643cc54af_isrForCrosstalkSources_1101465_43.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 0dc107515f2c0dfe1428cb9bec66818a parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/f5db92f8-5b63-46fc-8376-2696b496e84e_isrForCrosstalkSources_1101474_6.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/f5db92f8-5b63-46fc-8376-2696b496e84e_isrForCrosstalkSources_1101474_6.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash e26077297a4b09848ea047e0125ae4fc parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/7edda041-fc9f-4597-9a03-2caa813b4b93_isrForCrosstalkSources_1101468_59.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/7edda041-fc9f-4597-9a03-2caa813b4b93_isrForCrosstalkSources_1101468_59.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 102c2484e36db8064f2947c2cce70c0c parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/eedbead3-011d-474e-8e4b-1eb7e9d4b1bf_isrForCrosstalkSources_1101474_22.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/eedbead3-011d-474e-8e4b-1eb7e9d4b1bf_isrForCrosstalkSources_1101474_22.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash b99f5e3220bc4b30721832a94737ed17 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/47432b96-ca72-4876-ab00-358ddac5baa7_isrForCrosstalkSources_1101468_27.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/47432b96-ca72-4876-ab00-358ddac5baa7_isrForCrosstalkSources_1101468_27.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 752959034ccfe06ab252234c50a4f1db parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/5380299a-97ca-4c39-8aac-04266325188e_isrForCrosstalkSources_1101468_45.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/5380299a-97ca-4c39-8aac-04266325188e_isrForCrosstalkSources_1101468_45.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash da0a581431fef853aabd9a98d99b3a60 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/722d568f-b6e7-4ff6-95fb-e2421ce094a3_isrForCrosstalkSources_1101474_28.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/722d568f-b6e7-4ff6-95fb-e2421ce094a3_isrForCrosstalkSources_1101474_28.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash ce6234e9e839a1bf8294276e1dac4e77 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/c4d0b8ac-541d-4b72-b52c-e984f10f40af_isrForCrosstalkSources_1101474_39.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/c4d0b8ac-541d-4b72-b52c-e984f10f40af_isrForCrosstalkSources_1101474_39.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 833577b26507c83b48f455066d71dba4 parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/0bd6c4af-5e3e-454a-b2b8-85fc855e981e_isrForCrosstalkSources_1101474_18.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/0bd6c4af-5e3e-454a-b2b8-85fc855e981e_isrForCrosstalkSources_1101474_18.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 52d00d6e98e7a9479499c635f9ecec89 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/6575c630-ddee-4ccd-ac26-4dfbcd0ca1f2_isrForCrosstalkSources_1101471_18.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/6575c630-ddee-4ccd-ac26-4dfbcd0ca1f2_isrForCrosstalkSources_1101471_18.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 17b1f798e32ec32b1238ac26854097a7 parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/f76dc2e8-d41d-4059-8685-706603dc7f6f_isrForCrosstalkSources_1101465_6.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/f76dc2e8-d41d-4059-8685-706603dc7f6f_isrForCrosstalkSources_1101465_6.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 7e2d7f59b0e51c01e3fadf2c6cad5619 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/60ea1821-b012-486b-ad88-26dd73709904_isrForCrosstalkSources_1101471_2.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/60ea1821-b012-486b-ad88-26dd73709904_isrForCrosstalkSources_1101471_2.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 81421277ca076765e7bccfb1ba385519 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/43aa8481-1b1a-4c39-81c1-b6f6c43394a2_isrForCrosstalkSources_1101468_47.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/43aa8481-1b1a-4c39-81c1-b6f6c43394a2_isrForCrosstalkSources_1101468_47.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 5c7a0721243480970bbbc2b1b514e495 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/12fca3f4-d2b2-424d-895b-f1922a63c00e_isrForCrosstalkSources_1101474_43.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/12fca3f4-d2b2-424d-895b-f1922a63c00e_isrForCrosstalkSources_1101474_43.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 5b2372fff7319e1e43f95306fd403235 parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/bc0d1640-4e3e-49ae-b398-d03c93f15cdc_isrForCrosstalkSources_1101474_55.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/bc0d1640-4e3e-49ae-b398-d03c93f15cdc_isrForCrosstalkSources_1101474_55.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 72708528e9502138aa7932023c82d2ea parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/04b4c9fa-6db2-4cca-bdb5-81d2ee8c9e2a_isrForCrosstalkSources_1101465_19.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/04b4c9fa-6db2-4cca-bdb5-81d2ee8c9e2a_isrForCrosstalkSources_1101465_19.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 651bfdbdf529724bc77f3186b25d48c7 parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/557f7149-24d5-44d8-94af-a68f1fee19a5_isrForCrosstalkSources_1101471_12.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/557f7149-24d5-44d8-94af-a68f1fee19a5_isrForCrosstalkSources_1101471_12.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 829df29b2e8eee3d90a6fa2b7c145313 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/21a862b3-d64e-4f6a-8e61-817d637f2f27_isrForCrosstalkSources_1101471_4.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/21a862b3-d64e-4f6a-8e61-817d637f2f27_isrForCrosstalkSources_1101471_4.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 369285c0552012a65be31beb979bc238 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/cf3502a7-68e9-46ac-b93e-9dfc41b64cf0_isrForCrosstalkSources_1101474_50.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/cf3502a7-68e9-46ac-b93e-9dfc41b64cf0_isrForCrosstalkSources_1101474_50.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash c0da2834dafeb17510e30e8900f64584 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/47ea04d7-4b65-4d8a-b19b-419f6f428a24_isrForCrosstalkSources_1101465_28.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/47ea04d7-4b65-4d8a-b19b-419f6f428a24_isrForCrosstalkSources_1101465_28.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 1288d5acf7f398523a1fd64cb9a7dda4 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/bb63d8d8-a161-40cd-8912-97f0cdd6ebf4_isrForCrosstalkSources_1101474_61.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/bb63d8d8-a161-40cd-8912-97f0cdd6ebf4_isrForCrosstalkSources_1101474_61.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 2563125fe427b4b1f04bf159dadc8fb5 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/4aad3dd3-d6a6-4094-8c97-e72f7bd2c1d8_isrForCrosstalkSources_1101465_15.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/4aad3dd3-d6a6-4094-8c97-e72f7bd2c1d8_isrForCrosstalkSources_1101465_15.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 673d3a1de488963756275dfc625bd032 parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/aa7c637e-0e90-4233-a780-77eba01822c8_isrForCrosstalkSources_1101474_4.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/aa7c637e-0e90-4233-a780-77eba01822c8_isrForCrosstalkSources_1101474_4.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash f9da27a08c7ab147ab8e43fed7ca0076 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/3a5ffb9e-aa90-4166-a3c8-41bdb2b7b74e_isrForCrosstalkSources_1101471_10.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/3a5ffb9e-aa90-4166-a3c8-41bdb2b7b74e_isrForCrosstalkSources_1101471_10.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 89b0254a0309a33800f9a12ba35a99ed parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/ce41e099-f74e-4d21-88b0-80ff95faf956_isrForCrosstalkSources_1101465_3.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/ce41e099-f74e-4d21-88b0-80ff95faf956_isrForCrosstalkSources_1101465_3.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash e39c8931fbc9e46a4475960f12fd4915 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/00b92379-145f-42d5-85d1-eb7fbaef31ee_isrForCrosstalkSources_1101468_37.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/00b92379-145f-42d5-85d1-eb7fbaef31ee_isrForCrosstalkSources_1101468_37.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash e3146c224b555546f4fa85349284693b parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/2fe0418a-fb2f-4f5f-85cd-ceead2a2f193_isrForCrosstalkSources_1101468_4.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/2fe0418a-fb2f-4f5f-85cd-ceead2a2f193_isrForCrosstalkSources_1101468_4.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 6e0003b08b800f5ecdd3405cb790dbf7 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/6283eb8e-06f3-4b19-895a-963b2a09f3bc_isrForCrosstalkSources_1101468_14.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/6283eb8e-06f3-4b19-895a-963b2a09f3bc_isrForCrosstalkSources_1101468_14.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 26594b3c1af3094b55b2d4238682c45b parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/3086e792-ff3b-42ef-86b1-4dc27fbd811a_isrForCrosstalkSources_1101471_3.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/3086e792-ff3b-42ef-86b1-4dc27fbd811a_isrForCrosstalkSources_1101471_3.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash f76dd45a75ab8d3f2d116e687816a087 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/97dda7cf-5aae-4737-a3e6-1dc12bef8cc0_isrForCrosstalkSources_1101474_36.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/97dda7cf-5aae-4737-a3e6-1dc12bef8cc0_isrForCrosstalkSources_1101474_36.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 312e607aae331c00f12bf99ffb253966 parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/cc48e68c-be54-4a26-8521-fc6c53e0053e_isrForCrosstalkSources_1101474_51.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/cc48e68c-be54-4a26-8521-fc6c53e0053e_isrForCrosstalkSources_1101474_51.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 2ca932b389e86079ac63c7ad2dec358b parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/9002041b-11f9-4c07-83a9-9774c0cb5a46_isrForCrosstalkSources_1101474_48.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/9002041b-11f9-4c07-83a9-9774c0cb5a46_isrForCrosstalkSources_1101474_48.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash d8c72b849bd0feb2ef1af75a8c11f585 parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/38fcfa64-c923-4d69-9870-a789e986e74b_isrForCrosstalkSources_1101465_50.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/38fcfa64-c923-4d69-9870-a789e986e74b_isrForCrosstalkSources_1101465_50.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 6995bddfac0b6b48247f6a1527107339 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/f9376738-a2b3-4ddb-8a63-01aaa94c6b8f_isrForCrosstalkSources_1101471_54.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/f9376738-a2b3-4ddb-8a63-01aaa94c6b8f_isrForCrosstalkSources_1101471_54.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash fbd31f20c3a12223f7bfe36eda1d13df parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/68a6ecac-fffd-4790-b3cd-aa7ac87ac010_isrForCrosstalkSources_1101465_49.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/68a6ecac-fffd-4790-b3cd-aa7ac87ac010_isrForCrosstalkSources_1101465_49.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 8991ce4eec96388353cd7470b1c7e1b4 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/36fe95ec-ba29-46a3-a801-e331eec48f87_isrForCrosstalkSources_1101465_53.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/36fe95ec-ba29-46a3-a801-e331eec48f87_isrForCrosstalkSources_1101465_53.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 56796f390487e1567c1b6f674f1b5a54 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/99095763-3998-49bb-bdc9-60307b3414ae_isrForCrosstalkSources_1101468_52.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/99095763-3998-49bb-bdc9-60307b3414ae_isrForCrosstalkSources_1101468_52.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 8dcf6f083a7c10440b77294cb788460c parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/a4c97ae0-f7d1-4b7d-8622-80cd6fd6c49c_isrForCrosstalkSources_1101465_5.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/a4c97ae0-f7d1-4b7d-8622-80cd6fd6c49c_isrForCrosstalkSources_1101465_5.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 684e93369139559b91854b206e1233f7 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/0546b155-36d4-49c5-9aa3-df3021562e8e_isrForCrosstalkSources_1101471_62.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/0546b155-36d4-49c5-9aa3-df3021562e8e_isrForCrosstalkSources_1101471_62.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash de8ff57ec49e9e052d79f3b064a50f26 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/a528da25-034e-4e90-8171-0f3c1494660f_isrForCrosstalkSources_1101468_34.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/a528da25-034e-4e90-8171-0f3c1494660f_isrForCrosstalkSources_1101468_34.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash 21f5675624f290ed0410ad803637cbb1 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/9f466496-948e-4ac8-bd11-6e83fa9a3451_isrForCrosstalkSources_1101474_49.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/9f466496-948e-4ac8-bd11-6e83fa9a3451_isrForCrosstalkSources_1101474_49.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 63d94c0617f11eb3c6b6ced6f998c028 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/01ace463-65d9-47f3-8d39-3f212bdcb723_isrForCrosstalkSources_1101465_42.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/01ace463-65d9-47f3-8d39-3f212bdcb723_isrForCrosstalkSources_1101465_42.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 5686e11d665219940fbcd69982544851 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/225d0511-3083-442c-824c-5d420b42b755_isrForCrosstalkSources_1101465_46.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/225d0511-3083-442c-824c-5d420b42b755_isrForCrosstalkSources_1101465_46.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 4f5d06c3245ce575e379ed1c225492ae parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/2cf08127-5275-45be-9512-80d1ac559cc2_isrForCrosstalkSources_1101465_57.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/2cf08127-5275-45be-9512-80d1ac559cc2_isrForCrosstalkSources_1101465_57.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 2b6cac546084bf60e8d02e430e7ce0a4 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/b7e3a08b-9057-4b3e-a407-eda893c088a0_isrForCrosstalkSources_1101468_23.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/b7e3a08b-9057-4b3e-a407-eda893c088a0_isrForCrosstalkSources_1101468_23.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 37dbf83c610d904b752377087635c889 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/20d84bba-d561-4871-bc24-828bc5ebef2e_isrForCrosstalkSources_1101474_21.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/20d84bba-d561-4871-bc24-828bc5ebef2e_isrForCrosstalkSources_1101474_21.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 4ac4b0482f13e80f8b256852fbba7259 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/7d8ee1a7-4422-4598-b87e-aa64bea15bdc_isrForCrosstalkSources_1101468_32.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/7d8ee1a7-4422-4598-b87e-aa64bea15bdc_isrForCrosstalkSources_1101468_32.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash bd521218a216ca394ed4a6368896ae81 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/036154ca-d642-4c98-a577-72ae525d3601_isrForCrosstalkSources_1101468_15.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/036154ca-d642-4c98-a577-72ae525d3601_isrForCrosstalkSources_1101468_15.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 2a22f35c4a107a1f5297f49dc7a394ca parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/8d41c1df-f35e-4606-9517-3fef04cf1e74_isrForCrosstalkSources_1101471_61.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/8d41c1df-f35e-4606-9517-3fef04cf1e74_isrForCrosstalkSources_1101471_61.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 1b62f94aacd03b6807d12d712d5692c9 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/782434b0-daac-45ae-87a7-298c2a6bcf3d_isrForCrosstalkSources_1101465_52.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/782434b0-daac-45ae-87a7-298c2a6bcf3d_isrForCrosstalkSources_1101465_52.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash e1f3e9525adc073e657de650f74efeed parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/65f09ad8-5a08-4fa2-8258-aed131b483a7_isrForCrosstalkSources_1101465_4.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/65f09ad8-5a08-4fa2-8258-aed131b483a7_isrForCrosstalkSources_1101465_4.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash dda1d056b3357d3bcc96dd6c62b96f90 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/4057f1b4-5e86-441f-810c-5c72fa5d15c8_isrForCrosstalkSources_1101471_16.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/4057f1b4-5e86-441f-810c-5c72fa5d15c8_isrForCrosstalkSources_1101471_16.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 070aa13d051aff44a9f6ee7cc75485e3 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/9a114a1e-af9c-4ce7-8157-547db3117173_isrForCrosstalkSources_1101468_16.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/9a114a1e-af9c-4ce7-8157-547db3117173_isrForCrosstalkSources_1101468_16.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash ffd4cdd2edb74553065ddb38f5c33707 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/69053a71-4b5c-46b0-aa09-fabbe3908137_isrForCrosstalkSources_1101474_34.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/69053a71-4b5c-46b0-aa09-fabbe3908137_isrForCrosstalkSources_1101474_34.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 1704c0968c56218f12e114307153ae98 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/89f0fe78-60c7-446e-9d64-1e07e8962d43_isrForCrosstalkSources_1101471_8.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/89f0fe78-60c7-446e-9d64-1e07e8962d43_isrForCrosstalkSources_1101471_8.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash dee27fb0f008066be319faafb9b998ab parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/01cc464d-1c9c-4f97-a512-abb8c006021d_isrForCrosstalkSources_1101471_57.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/01cc464d-1c9c-4f97-a512-abb8c006021d_isrForCrosstalkSources_1101471_57.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 42d007bfc0bf543c8aa55c1d6b8c6467 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/482908ac-6494-4a64-b33e-495187008ba7_isrForCrosstalkSources_1101465_26.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/482908ac-6494-4a64-b33e-495187008ba7_isrForCrosstalkSources_1101465_26.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash c403c73574ab0cce5b56a45ecb830aa5 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/259cc124-e11e-4dae-8011-a6c9bc7cab64_isrForCrosstalkSources_1101468_36.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/259cc124-e11e-4dae-8011-a6c9bc7cab64_isrForCrosstalkSources_1101468_36.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 88ece6163fb105cbc5f83b22837e340b parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/523a2f26-f1b6-4938-b25e-b60433150f09_isrForCrosstalkSources_1101468_44.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/523a2f26-f1b6-4938-b25e-b60433150f09_isrForCrosstalkSources_1101468_44.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 37b22db0d03493219939e00d52d8e625 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/549bb2a6-6d3e-4ad9-8db3-b58eeec93404_isrForCrosstalkSources_1101465_56.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/549bb2a6-6d3e-4ad9-8db3-b58eeec93404_isrForCrosstalkSources_1101465_56.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 6cabaf4469b2c8cf3c18c821c3e16186 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/9740d960-b94f-4b67-bf91-683995b39183_isrForCrosstalkSources_1101471_22.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/9740d960-b94f-4b67-bf91-683995b39183_isrForCrosstalkSources_1101471_22.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 510f18651658e9f22e4b6255b5ed1ef5 parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/903a7008-c2a8-4daa-9c10-df2a909b036f_isrForCrosstalkSources_1101471_27.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/903a7008-c2a8-4daa-9c10-df2a909b036f_isrForCrosstalkSources_1101471_27.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 01d34d61779f089433285c452a15e01a parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/5890ae21-0327-4b2c-b5e9-9681764cd56d_isrForCrosstalkSources_1101465_45.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/5890ae21-0327-4b2c-b5e9-9681764cd56d_isrForCrosstalkSources_1101465_45.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash a708313ff971fb2734aaa6c5b1f6b85b parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/647ed4cd-7f35-4e3b-8ef3-51ef3c2e62f3_isrForCrosstalkSources_1101474_8.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/647ed4cd-7f35-4e3b-8ef3-51ef3c2e62f3_isrForCrosstalkSources_1101474_8.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 3ebcd97ff1c0f3576a7a914675ab7cbc parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/0586009a-1350-4dfd-a341-7e37197cd4fc_isrForCrosstalkSources_1101474_30.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/0586009a-1350-4dfd-a341-7e37197cd4fc_isrForCrosstalkSources_1101474_30.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash f46225fb994bbceb0c5e3e1bcbce1814 parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/fee0277c-ef15-4385-8180-9b8b98d5147f_isrForCrosstalkSources_1101471_6.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/fee0277c-ef15-4385-8180-9b8b98d5147f_isrForCrosstalkSources_1101471_6.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash d1a595b0ff3eeff0f58eeee0891209e7 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/1ab10ea8-8f62-4b10-a9ba-10ec761d4464_isrForCrosstalkSources_1101471_53.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/1ab10ea8-8f62-4b10-a9ba-10ec761d4464_isrForCrosstalkSources_1101471_53.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 175050acdf839408929bb7dc199e86df parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/4abd2c0c-e7ca-4f94-bf19-fa5645a42838_isrForCrosstalkSources_1101474_44.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/4abd2c0c-e7ca-4f94-bf19-fa5645a42838_isrForCrosstalkSources_1101474_44.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash f8b80cf7420289881cbf244cf95d6a73 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/1334c6b8-9498-4e38-aa76-a37b2a8a4127_isrForCrosstalkSources_1101468_38.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/1334c6b8-9498-4e38-aa76-a37b2a8a4127_isrForCrosstalkSources_1101468_38.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash b8a3042b6f5cf6f83ca2e74d8e76dbee parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/f7c34b97-759f-486e-b768-7fa7cae9e33c_isrForCrosstalkSources_1101474_46.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/f7c34b97-759f-486e-b768-7fa7cae9e33c_isrForCrosstalkSources_1101474_46.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash e0eee7e23561e27223ccb6210de7d4c1 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/492d9121-fa1d-4c0e-a542-990a391c8e6e_isrForCrosstalkSources_1101471_30.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/492d9121-fa1d-4c0e-a542-990a391c8e6e_isrForCrosstalkSources_1101471_30.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 72bc551c69695acc5fc1698c820cba15 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/accfbb72-aaa2-4a81-b43c-e63ada5a9cbf_isrForCrosstalkSources_1101465_61.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/accfbb72-aaa2-4a81-b43c-e63ada5a9cbf_isrForCrosstalkSources_1101465_61.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash bcc4ef207ba4a4db44985d536bb6fa10 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/e8b4dd7b-6a8f-460d-acbc-1cbd4fe8ee51_isrForCrosstalkSources_1101474_25.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/e8b4dd7b-6a8f-460d-acbc-1cbd4fe8ee51_isrForCrosstalkSources_1101474_25.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 0d1a2553f9a217d7d83e4ed25cd04856 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/582d8fd0-16cd-46df-88ac-29ad2430e9e7_isrForCrosstalkSources_1101471_52.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/582d8fd0-16cd-46df-88ac-29ad2430e9e7_isrForCrosstalkSources_1101471_52.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash def362ded478807539e8734fe6d8ba18 parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/4b69004d-d99e-4634-a04c-843ab62d2141_isrForCrosstalkSources_1101468_30.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/4b69004d-d99e-4634-a04c-843ab62d2141_isrForCrosstalkSources_1101468_30.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash bbf2829d8f99ff97c9fd4d3c9fbcaed9 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/6e43fe3d-c04c-4757-91c2-48f9e4c60a23_isrForCrosstalkSources_1101471_24.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/6e43fe3d-c04c-4757-91c2-48f9e4c60a23_isrForCrosstalkSources_1101471_24.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 44e4ddddc2c0fc4876291cdf4a50267d parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/c2b6064c-5abe-4cd9-b3ce-1d766e6f61fd_isrForCrosstalkSources_1101468_2.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/c2b6064c-5abe-4cd9-b3ce-1d766e6f61fd_isrForCrosstalkSources_1101468_2.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 8882f7864561cf515b308a678f0845f0 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/42c206e0-1229-485d-b964-2a233b519601_isrForCrosstalkSources_1101474_12.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/42c206e0-1229-485d-b964-2a233b519601_isrForCrosstalkSources_1101474_12.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 58de3aea9b3f242325bdbea87a14d97f parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/a4105d02-01bc-4e75-abab-e2c599a7d52b_isrForCrosstalkSources_1101474_54.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/a4105d02-01bc-4e75-abab-e2c599a7d52b_isrForCrosstalkSources_1101474_54.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 5c4accf1841c1b567856b712d457c7a6 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/5ce09d38-3a05-4424-97db-e310e7563dc6_isrForCrosstalkSources_1101465_48.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/5ce09d38-3a05-4424-97db-e310e7563dc6_isrForCrosstalkSources_1101465_48.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 032bc453c8cf0f0d2c8d16bbb15e96f5 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/20fe314a-6113-4037-a3d6-d7d1ee2cc33e_isrForCrosstalkSources_1101468_26.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/20fe314a-6113-4037-a3d6-d7d1ee2cc33e_isrForCrosstalkSources_1101468_26.stderr parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 50285f6496a77866f1d80323de5da755 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498ba4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/701d4dd4-421a-4c73-b636-54ffac82f949_isrForCrosstalkSources_1101474_13.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/701d4dd4-421a-4c73-b636-54ffac82f949_isrForCrosstalkSources_1101474_13.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 67306bcc5da85ca154ab7b6006a553f7 parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/8ae13032-c98a-4d27-a126-b9c1f3b66da6_isrForCrosstalkSources_1101465_37.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/8ae13032-c98a-4d27-a126-b9c1f3b66da6_isrForCrosstalkSources_1101465_37.stderr parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash b97eecfbfce384e722fce6e09f3a08d1 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/f922b3e9-1763-4e82-8c4b-5db1f00ef4b3_isrForCrosstalkSources_1101468_42.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/f922b3e9-1763-4e82-8c4b-5db1f00ef4b3_isrForCrosstalkSources_1101468_42.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash b41f3e923e817743c0ab63212b29fde9 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/5179c732-2584-4925-80ca-7e71aa85d1ca_isrForCrosstalkSources_1101474_31.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/5179c732-2584-4925-80ca-7e71aa85d1ca_isrForCrosstalkSources_1101474_31.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash d82dd32d2fdd53e4503e931f19dda8a1 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/38927832-d5fa-477e-b0b2-ed5e4f2f9907_isrForCrosstalkSources_1101465_55.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/38927832-d5fa-477e-b0b2-ed5e4f2f9907_isrForCrosstalkSources_1101465_55.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash cdff33c37a1c830984bb23c28eb2d43a parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/22bd586f-eaf1-4aae-bd47-b3749db0bf91_isrForCrosstalkSources_1101471_36.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/22bd586f-eaf1-4aae-bd47-b3749db0bf91_isrForCrosstalkSources_1101471_36.stderr parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash a769650cc60cf0668827ea59ee72d5dc parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/47deb65e-c865-4814-a00e-9cfc903e025b_isrForCrosstalkSources_1101465_10.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/47deb65e-c865-4814-a00e-9cfc903e025b_isrForCrosstalkSources_1101465_10.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 812c18e9785b68511fd6cd4214080d76 parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/3679b6db-77d9-48e9-85cf-510fb205cfbb_isrForCrosstalkSources_1101465_34.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/3679b6db-77d9-48e9-85cf-510fb205cfbb_isrForCrosstalkSources_1101465_34.stderr parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash d3d902ca90fb36294d213f636effd7bf parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/5d091e6f-4ebd-41ce-8e9f-af7eeab0f090_isrForCrosstalkSources_1101465_8.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/5d091e6f-4ebd-41ce-8e9f-af7eeab0f090_isrForCrosstalkSources_1101465_8.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 4f5fba32e1cd10faa88bb60230fa7caa parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/1cbbd28b-1dc5-4d8f-8322-0c6ba858349d_isrForCrosstalkSources_1101468_43.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/1cbbd28b-1dc5-4d8f-8322-0c6ba858349d_isrForCrosstalkSources_1101468_43.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash ec582020d17bc06ca817a6dc20ccd0fb parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/4266bc6d-d2d6-4481-8143-be12b70b850b_isrForCrosstalkSources_1101471_11.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/4266bc6d-d2d6-4481-8143-be12b70b850b_isrForCrosstalkSources_1101471_11.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash c443f88208ac4072ace445f071a9f3db parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/55f8f94d-10fe-432e-8b0c-1e4c2100719f_isrForCrosstalkSources_1101471_59.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/55f8f94d-10fe-432e-8b0c-1e4c2100719f_isrForCrosstalkSources_1101471_59.stderr parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 3bd6c95d4d5f6be2da3cdcef05f36edb parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/fdc2501e-e61c-46ee-9553-3d5069604d75_isrForCrosstalkSources_1101465_59.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/fdc2501e-e61c-46ee-9553-3d5069604d75_isrForCrosstalkSources_1101465_59.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 040609dc965fef06eaf7cfd9136f0fe4 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/f44b483d-6bdf-4d1d-a864-e396f67d8201_isrForCrosstalkSources_1101465_20.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/f44b483d-6bdf-4d1d-a864-e396f67d8201_isrForCrosstalkSources_1101465_20.stderr parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 0d6691598304c5d3b93058ac66b02e85 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/81ac2801-b35c-48d7-97ae-4077cc391f22_isrForCrosstalkSources_1101465_24.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/81ac2801-b35c-48d7-97ae-4077cc391f22_isrForCrosstalkSources_1101465_24.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash 9845013f5f3da5789f5bfd52f4e8c314 parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/ab8ade4f-3644-4a02-832e-0e9ad1597cef_isrForCrosstalkSources_1101471_20.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/ab8ade4f-3644-4a02-832e-0e9ad1597cef_isrForCrosstalkSources_1101471_20.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash e1470d7e1e8b63e67f74a5f4e2102679 parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/75271aa6-adf2-4965-a288-82e0247a15e4_isrForCrosstalkSources_1101474_19.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/75271aa6-adf2-4965-a288-82e0247a15e4_isrForCrosstalkSources_1101474_19.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 45018b781a8a4e19443619ad3a611f90 parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/cbedec63-accf-4811-9c01-d1f8ac868672_isrForCrosstalkSources_1101465_54.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/cbedec63-accf-4811-9c01-d1f8ac868672_isrForCrosstalkSources_1101465_54.stderr parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash ee550d709272eb86259349c154210dfd parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/a4860be9-da49-4e97-a086-0fcbac27ae4c_isrForCrosstalkSources_1101471_37.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/a4860be9-da49-4e97-a086-0fcbac27ae4c_isrForCrosstalkSources_1101471_37.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash a4f130ae4c17861f9b374868171a50f3 parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/5dc5bfcf-f106-4b8a-9038-3d353714bec7_isrForCrosstalkSources_1101471_47.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/5dc5bfcf-f106-4b8a-9038-3d353714bec7_isrForCrosstalkSources_1101471_47.stderr parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash a51476e464e6ac1c18353ecd8094b4ab parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/ec5a4bdd-ce36-4184-87f1-d0432eac7ad7_isrForCrosstalkSources_1101468_46.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/ec5a4bdd-ce36-4184-87f1-d0432eac7ad7_isrForCrosstalkSources_1101468_46.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash a9b5ae2dfeda627f61ee1830b5e3ffc1 parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/c512f342-da05-4acf-8557-05c005e2b08f_isrForCrosstalkSources_1101471_21.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/c512f342-da05-4acf-8557-05c005e2b08f_isrForCrosstalkSources_1101471_21.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 70ad9c6d5da3f8e3f57e1ff00fc94a77 parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/683411b6-dae4-45ce-be15-6a428032cab1_isrForCrosstalkSources_1101474_42.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/683411b6-dae4-45ce-be15-6a428032cab1_isrForCrosstalkSources_1101474_42.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash e56dc08065f2ba98b34b8e579a0db148 parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/61eab9f2-22e1-492a-9ca9-e8c1f474f1d6_isrForCrosstalkSources_1101474_53.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/61eab9f2-22e1-492a-9ca9-e8c1f474f1d6_isrForCrosstalkSources_1101474_53.stderr parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash fa9d62d355e01b7b228c8de24464cff2 parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/d8efc727-5cb4-45a1-b86c-3ab34826ee05_isrForCrosstalkSources_1101471_35.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/d8efc727-5cb4-45a1-b86c-3ab34826ee05_isrForCrosstalkSources_1101471_35.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash e665981d72604b11e603d4e8d68c060f parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/ebef071b-4c2b-4317-bb3d-e8c17debd107_isrForCrosstalkSources_1101474_5.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/ebef071b-4c2b-4317-bb3d-e8c17debd107_isrForCrosstalkSources_1101474_5.stderr parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash d140cb744277f68d604ddab150bdaeae parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/a7bcd230-bf57-453c-9e39-7c12dd37a53e_isrForCrosstalkSources_1101474_3.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/a7bcd230-bf57-453c-9e39-7c12dd37a53e_isrForCrosstalkSources_1101474_3.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 56a3ac61a747de9a159971a96969b8fc parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/0c99a475-37d5-4c78-8b85-340f61f13789_isrForCrosstalkSources_1101471_40.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/0c99a475-37d5-4c78-8b85-340f61f13789_isrForCrosstalkSources_1101471_40.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 00cfdfe816eef1f685b5811e2db49471 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/a89be5eb-1228-4cdc-9553-c22af674fc43_isrForCrosstalkSources_1101471_50.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/a89be5eb-1228-4cdc-9553-c22af674fc43_isrForCrosstalkSources_1101471_50.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 3dcfb1af56c3064b27d71b58d203600d parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/a984c62e-ebad-4c15-b6c9-bc2be72e5079_isrForCrosstalkSources_1101471_56.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/a984c62e-ebad-4c15-b6c9-bc2be72e5079_isrForCrosstalkSources_1101471_56.stderr parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 885e5f176ed1086d591e34c28d6d84b7 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/69070fa0-10cd-4a5f-93e1-73997fbcc03d_isrForCrosstalkSources_1101471_1.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/69070fa0-10cd-4a5f-93e1-73997fbcc03d_isrForCrosstalkSources_1101471_1.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 46ae3ef52c2e2379e12a253c24f2d399 parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/71a15409-e386-4453-91d7-a5d4dac5529c_isrForCrosstalkSources_1101474_41.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/71a15409-e386-4453-91d7-a5d4dac5529c_isrForCrosstalkSources_1101474_41.stderr parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash a27d9208d0b97961e489a97a971fd568 parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/06a2f99d-9c0a-46d9-ab09-7023dcdd6c86_isrForCrosstalkSources_1101474_23.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/06a2f99d-9c0a-46d9-ab09-7023dcdd6c86_isrForCrosstalkSources_1101474_23.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: 2024-03-19 00:39:24 proc_lsst.multi:146 [INFO] found job 96275 in provider local proc_lsst.multi INFO: found job 96275 in provider local parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 18c175d01d33fb3f5dbe989db7b2e260 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/d3a15c41-5791-4b06-9976-e0d1d22c80f0_isrForCrosstalkSources_1101468_31.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/d3a15c41-5791-4b06-9976-e0d1d22c80f0_isrForCrosstalkSources_1101468_31.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash b3134d7d8d65b3d3c60a768ea445cf8c parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/a226bb6c-cd76-4614-adc7-45ca5c6d4091_isrForCrosstalkSources_1101465_35.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/a226bb6c-cd76-4614-adc7-45ca5c6d4091_isrForCrosstalkSources_1101465_35.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 8abf748f8fdaac59b956ae45cde5345d parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/edc9a63a-430d-4b7b-8e95-7615e2c9534b_isrForCrosstalkSources_1101465_23.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/edc9a63a-430d-4b7b-8e95-7615e2c9534b_isrForCrosstalkSources_1101465_23.stderr parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 9f71a013c89fbee6973dc0224bf03299 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/752dd2f1-6a80-4bed-88e3-70f274f3cbdd_isrForCrosstalkSources_1101471_55.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/752dd2f1-6a80-4bed-88e3-70f274f3cbdd_isrForCrosstalkSources_1101471_55.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 7c27a66b2535a8db54be78270913c46b parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/b705e6c1-ecac-4e0a-b1f3-580b7e4400e4_isrForCrosstalkSources_1101468_9.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/b705e6c1-ecac-4e0a-b1f3-580b7e4400e4_isrForCrosstalkSources_1101468_9.stderr parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 64ac6393f98c1b09dc647c2aa8487860 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/2ec4d6d1-3edb-4d0a-9ef0-59cf3163b308_isrForCrosstalkSources_1101468_19.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/2ec4d6d1-3edb-4d0a-9ef0-59cf3163b308_isrForCrosstalkSources_1101468_19.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash f782bcf8fcbedefac7bc823674027a5e parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/22a7d9b5-2185-4c34-90c5-9bb52ca6afbb_isrForCrosstalkSources_1101474_16.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/22a7d9b5-2185-4c34-90c5-9bb52ca6afbb_isrForCrosstalkSources_1101474_16.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 22566847265b0bb45c60d3e9fb34b6d1 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/a993ba8a-f991-4b24-83ee-89e82408361c_isrForCrosstalkSources_1101474_40.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/a993ba8a-f991-4b24-83ee-89e82408361c_isrForCrosstalkSources_1101474_40.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 205ecc6cb2ad675ad19dfa84ad820324 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/80eb1844-0b61-4589-b727-52575fa13afe_isrForCrosstalkSources_1101471_25.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/80eb1844-0b61-4589-b727-52575fa13afe_isrForCrosstalkSources_1101471_25.stderr parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash be085e6cc3c9d56b502fe8db55412012 parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 165 parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/5fad046e-3002-419a-91f4-22ed064c427f_isrForCrosstalkSources_1101474_59.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/5fad046e-3002-419a-91f4-22ed064c427f_isrForCrosstalkSources_1101474_59.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 64d8c654bacea4ce69d6f198f4750937 parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/31895b69-bd81-44db-9714-cbda07db3486_isrForCrosstalkSources_1101474_7.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/31895b69-bd81-44db-9714-cbda07db3486_isrForCrosstalkSources_1101474_7.stderr parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 6f653252901bb7f794bdb368743beeb0 parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: Executor multi has 165 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 41 more blocks parsl.executors.status_handling INFO: Scaling out by 41 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-19 00:39:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-19 00:39:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-19 00:39:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/cb0b5f46-9029-498e-bbfb-ec69861340c8_isrForCrosstalkSources_1101474_60.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/cb0b5f46-9029-498e-bbfb-ec69861340c8_isrForCrosstalkSources_1101474_60.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash f2898d9769eeacea01b158aec5bcab89 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/b66d41a4-d165-4b2d-a8fb-85e28b5435d7_isrForCrosstalkSources_1101465_13.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/b66d41a4-d165-4b2d-a8fb-85e28b5435d7_isrForCrosstalkSources_1101465_13.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash e83e777cbb1ac51fa55063385fe7de2f parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/cae7b685-5066-40a7-8254-a2181d981804_isrForCrosstalkSources_1101465_40.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/cae7b685-5066-40a7-8254-a2181d981804_isrForCrosstalkSources_1101465_40.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 99febddd9526120dd0997251e87bfbae parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/4a4efa3f-e8ea-492e-9dcb-9d6d4c81d5eb_isrForCrosstalkSources_1101471_34.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/4a4efa3f-e8ea-492e-9dcb-9d6d4c81d5eb_isrForCrosstalkSources_1101471_34.stderr parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 55663dea15b224e94640d7c2c9cf8bac parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/48aad19f-1a13-41b0-8a14-7ce83dc89be0_isrForCrosstalkSources_1101471_13.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/48aad19f-1a13-41b0-8a14-7ce83dc89be0_isrForCrosstalkSources_1101471_13.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash db0acdc6edabb10f41c9c0181c8ce600 parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/a1547728-e8b8-4aad-bd5e-722430704fe3_isrForCrosstalkSources_1101474_47.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/a1547728-e8b8-4aad-bd5e-722430704fe3_isrForCrosstalkSources_1101474_47.stderr parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash ec4b8ce5b29e8f34f7f69d9953a3de79 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/9cd03a18-34c9-437c-a4e2-1a6f8324376d_isrForCrosstalkSources_1101471_32.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/9cd03a18-34c9-437c-a4e2-1a6f8324376d_isrForCrosstalkSources_1101471_32.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 7d2bd9a8a5ff75e17ae68a7c92772ebe parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/7a1ea6be-280d-4fed-bafe-17e926d73bee_isrForCrosstalkSources_1101465_14.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/7a1ea6be-280d-4fed-bafe-17e926d73bee_isrForCrosstalkSources_1101465_14.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 92a7991bfd4cb926c423a1ca7438b12e parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/dceac4e5-bf3f-4ade-ab07-1f11b1dc5a4f_isrForCrosstalkSources_1101468_24.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/dceac4e5-bf3f-4ade-ab07-1f11b1dc5a4f_isrForCrosstalkSources_1101468_24.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 7323a75a85ba98ec1dfdfe8ae586bbb2 parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/91de3360-d6d5-4cdc-b738-30d576628673_isrForCrosstalkSources_1101468_54.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/91de3360-d6d5-4cdc-b738-30d576628673_isrForCrosstalkSources_1101468_54.stderr parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 3a17613dfd389a0f2d8ee3b137fdba64 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/c1b7471e-58f6-4e96-be99-90e337f76297_isrForCrosstalkSources_1101468_10.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/c1b7471e-58f6-4e96-be99-90e337f76297_isrForCrosstalkSources_1101468_10.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 99d2dc9ec16895a9035c9e424c831ba2 parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/44090055-d331-41c9-a220-cb54648e2321_isrForCrosstalkSources_1101471_23.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/44090055-d331-41c9-a220-cb54648e2321_isrForCrosstalkSources_1101471_23.stderr parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash ad75176afa5ceeda90790576bbcf4dad parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/b008ab9b-b459-4edf-a4b4-0d2a0d160257_isrForCrosstalkSources_1101474_10.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/b008ab9b-b459-4edf-a4b4-0d2a0d160257_isrForCrosstalkSources_1101474_10.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 44516ad9a75680084404675704b892f8 parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/1c313475-e32f-4f3c-b257-0a4c1baaab95_isrForCrosstalkSources_1101465_62.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/1c313475-e32f-4f3c-b257-0a4c1baaab95_isrForCrosstalkSources_1101465_62.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 83cc4d7337b4f35d0fa06179516f8c99 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/96d9ecd5-8430-4840-bf7b-b4d5d69d7352_isrForCrosstalkSources_1101468_41.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/96d9ecd5-8430-4840-bf7b-b4d5d69d7352_isrForCrosstalkSources_1101468_41.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 8661c692d7d161947366b5a4df60f0d2 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/65b60993-b6fb-4795-8af9-e9d571ed0cbd_isrForCrosstalkSources_1101471_39.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/65b60993-b6fb-4795-8af9-e9d571ed0cbd_isrForCrosstalkSources_1101471_39.stderr parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 2b26a8491f46065543bef940faeb7805 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/31bfff51-c18d-4623-bb51-270de67adc56_isrForCrosstalkSources_1101468_1.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/31bfff51-c18d-4623-bb51-270de67adc56_isrForCrosstalkSources_1101468_1.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash 4e57b05315821017c4870c6cb904007c parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/1da1ee71-2cdc-436d-9fff-093d6fbce011_isrForCrosstalkSources_1101474_37.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/1da1ee71-2cdc-436d-9fff-093d6fbce011_isrForCrosstalkSources_1101474_37.stderr parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 67854befaa9700d05c3c5f6a0b9fe1d9 parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/0c5868dc-f790-4be4-95bd-bbcd17ce6a60_isrForCrosstalkSources_1101468_18.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/0c5868dc-f790-4be4-95bd-bbcd17ce6a60_isrForCrosstalkSources_1101468_18.stderr parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash 667b52e40062f5f5255a1786578502be parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/aa605f35-8d39-4bb3-a42e-155a50e9d3c2_isrForCrosstalkSources_1101471_29.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/aa605f35-8d39-4bb3-a42e-155a50e9d3c2_isrForCrosstalkSources_1101471_29.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 274c71b0fa117efa3f280a6e0d88f042 parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/5a5828aa-4632-4d95-8e40-b9a95176f6c8_isrForCrosstalkSources_1101471_60.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/5a5828aa-4632-4d95-8e40-b9a95176f6c8_isrForCrosstalkSources_1101471_60.stderr parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash f5c14e3f19807a0c2cee3996d6ec5eb5 parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/6e65604b-3861-4047-9d21-6e6ae5da7f34_isrForCrosstalkSources_1101468_56.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/6e65604b-3861-4047-9d21-6e6ae5da7f34_isrForCrosstalkSources_1101468_56.stderr parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash 7a5978082fa16c66e9c4af3d21829c14 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/e7224858-1ee7-4866-87aa-56663677221d_isrForCrosstalkSources_1101468_13.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/e7224858-1ee7-4866-87aa-56663677221d_isrForCrosstalkSources_1101468_13.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash c3d420e3c374c6101b6405383885788f parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/16491f04-ce38-4ab0-acfe-837b010b9805_isrForCrosstalkSources_1101468_25.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/16491f04-ce38-4ab0-acfe-837b010b9805_isrForCrosstalkSources_1101468_25.stderr parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash 8236a1e26cad853926c9289deb78ba85 parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/31ee2544-d886-4255-bde5-2a80679a8ad5_isrForCrosstalkSources_1101468_33.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/31ee2544-d886-4255-bde5-2a80679a8ad5_isrForCrosstalkSources_1101468_33.stderr parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 0737b97a202e77ace83a1b0a6d2084ea parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-19 00:39:24 proc_lsst.multi:170 [INFO] [multi] job_id 17225950 proc_lsst.multi INFO: [multi] job_id 17225950 2024-03-19 00:39:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:24 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225950 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225950 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17225950 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-19 00:39:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-19 00:39:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/4043b8d8-0173-4bb8-be14-f2e787f773be_isrForCrosstalkSources_1101465_18.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/4043b8d8-0173-4bb8-be14-f2e787f773be_isrForCrosstalkSources_1101465_18.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash 5464f3a086d936be6970370adca4a9eb parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/73f59019-17ec-46c9-ad97-273bfdc4372c_isrForCrosstalkSources_1101468_50.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/73f59019-17ec-46c9-ad97-273bfdc4372c_isrForCrosstalkSources_1101468_50.stderr parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash 36fa9994ff5087eca35129ba06af67da parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/d43d3882-9d6e-421d-9913-7310d4a24a79_isrForCrosstalkSources_1101471_28.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/d43d3882-9d6e-421d-9913-7310d4a24a79_isrForCrosstalkSources_1101471_28.stderr parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 101790b7dc2eb1a688f0018cec865508 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/962ba212-e1df-4477-89c8-a7829d017fbb_isrForCrosstalkSources_1101471_33.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/962ba212-e1df-4477-89c8-a7829d017fbb_isrForCrosstalkSources_1101471_33.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash da205d5ffa25f3e752a060dea63522b5 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/f1cf5f9d-11c9-499a-943c-52ea1738788c_isrForCrosstalkSources_1101474_57.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/f1cf5f9d-11c9-499a-943c-52ea1738788c_isrForCrosstalkSources_1101474_57.stderr parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash 7667c18dbced6066f085b014d28646e2 parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/8c9c5b41-94f0-4453-a9a7-341b193734dd_isrForCrosstalkSources_1101465_44.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/8c9c5b41-94f0-4453-a9a7-341b193734dd_isrForCrosstalkSources_1101465_44.stderr parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash bac573818a60d7fa2aca68755c62fb21 parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/6916ace6-1003-4d30-996e-074f2b79a201_isrForCrosstalkSources_1101468_60.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/6916ace6-1003-4d30-996e-074f2b79a201_isrForCrosstalkSources_1101468_60.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 6c5fae4914c05eb7af5a96fb0d3c7ec3 parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 203 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/6bced9ee-24f8-4d23-870e-6582c73f426b_isrForCrosstalkSources_1101468_6.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/6bced9ee-24f8-4d23-870e-6582c73f426b_isrForCrosstalkSources_1101468_6.stderr parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 71766837f67c2016f659a7dce15329e7 parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/5dc00973-54a8-4828-a3f1-7cdeed7f5648_isrForCrosstalkSources_1101474_29.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/5dc00973-54a8-4828-a3f1-7cdeed7f5648_isrForCrosstalkSources_1101474_29.stderr parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 5eaf21a634521aee96262b78d01172f2 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498076700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/1fcffecb-6536-4f9e-a1bf-a6d4848736a8_isrForCrosstalkSources_1101465_27.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/1fcffecb-6536-4f9e-a1bf-a6d4848736a8_isrForCrosstalkSources_1101465_27.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 920711a5906a9f93a9e6dc51bdde051f parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498076660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/e2f9340a-2960-47a1-a420-0249bdd01261_isrForCrosstalkSources_1101468_5.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/e2f9340a-2960-47a1-a420-0249bdd01261_isrForCrosstalkSources_1101468_5.stderr parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 408f60b4ab53c3fa4749410df62a5a77 parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/c6f0416f-d221-405f-a81e-162657cd967d_isrForCrosstalkSources_1101474_14.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/c6f0416f-d221-405f-a81e-162657cd967d_isrForCrosstalkSources_1101474_14.stderr parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 0f127f5091847dee4fd77e507147e8ec parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/74ea063a-ab74-466e-96db-3b7028a5641b_isrForCrosstalkSources_1101468_7.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/74ea063a-ab74-466e-96db-3b7028a5641b_isrForCrosstalkSources_1101468_7.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash b64320bfa6277664bed32f329f6f8aca parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/20126102-45f1-4025-a1ed-db1e7925ba79_isrForCrosstalkSources_1101471_26.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/20126102-45f1-4025-a1ed-db1e7925ba79_isrForCrosstalkSources_1101471_26.stderr parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash 33bf038f8293b863819996f963e46f70 parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/d8b6a722-d25d-48fd-973a-e8930822f98f_isrForCrosstalkSources_1101468_61.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/d8b6a722-d25d-48fd-973a-e8930822f98f_isrForCrosstalkSources_1101468_61.stderr parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash c39334c6d9180add580382334ed70e48 parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/b96fb477-7999-4a18-9c3d-d64a7d0b06bf_isrForCrosstalkSources_1101474_27.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/b96fb477-7999-4a18-9c3d-d64a7d0b06bf_isrForCrosstalkSources_1101474_27.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash d85b303cb8829b163449d42123839578 parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/4946a7e2-a828-4c0c-8c26-fd225f110141_isrForCrosstalkSources_1101465_17.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/4946a7e2-a828-4c0c-8c26-fd225f110141_isrForCrosstalkSources_1101465_17.stderr parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash 141541d7fbbf7c8b01149232089891d6 parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/3a73c9f9-3e61-47b2-a2cc-2ea6e9ee3a14_isrForCrosstalkSources_1101468_35.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/3a73c9f9-3e61-47b2-a2cc-2ea6e9ee3a14_isrForCrosstalkSources_1101468_35.stderr parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 6c0429706ce0c92c878283a7584802f5 parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/fdbb5efb-36e4-4d58-8f86-055a5c6182b6_isrForCrosstalkSources_1101468_3.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/fdbb5efb-36e4-4d58-8f86-055a5c6182b6_isrForCrosstalkSources_1101468_3.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash 0068684bad484b17ae0e700655220bf6 parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/92924a35-1721-4bac-b58a-aba06e5d9433_isrForCrosstalkSources_1101465_60.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/92924a35-1721-4bac-b58a-aba06e5d9433_isrForCrosstalkSources_1101465_60.stderr parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash e3c5f54fbf0e39e7f6675e0649d12354 parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/2ae58f82-10bc-4f35-b013-e1c16c920e0f_isrForCrosstalkSources_1101474_1.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/2ae58f82-10bc-4f35-b013-e1c16c920e0f_isrForCrosstalkSources_1101474_1.stderr parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash 9edcf65488bb43ea38d2b4acc62deb17 parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/1b2aa3ba-af89-4b41-9879-a6b906ea41b5_isrForCrosstalkSources_1101468_39.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/1b2aa3ba-af89-4b41-9879-a6b906ea41b5_isrForCrosstalkSources_1101468_39.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 618e831a9f83b1c962f43440b4ee3e83 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/e224e802-0e48-43f9-957a-508a9d420bc6_isrForCrosstalkSources_1101465_32.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/e224e802-0e48-43f9-957a-508a9d420bc6_isrForCrosstalkSources_1101465_32.stderr parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 25e48ef147b45d2badbb8ee970609c45 parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/82ddc326-6143-41a4-9ac3-3fd8e6a99087_isrForCrosstalkSources_1101474_17.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/82ddc326-6143-41a4-9ac3-3fd8e6a99087_isrForCrosstalkSources_1101474_17.stderr parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash fed8f0d8241451e44b492d7ca54052fc parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/1ecef981-38d7-4275-a920-23f998695a19_isrForCrosstalkSources_1101465_31.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/1ecef981-38d7-4275-a920-23f998695a19_isrForCrosstalkSources_1101465_31.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 94b664b28dcc0145bc404d91643219c7 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/12d63abf-e6f0-4f5a-aa91-ae7af215e0f6_isrForCrosstalkSources_1101474_2.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/12d63abf-e6f0-4f5a-aa91-ae7af215e0f6_isrForCrosstalkSources_1101474_2.stderr parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash 6c6d3d1d64b6bfd224d11231ed84f8d6 parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/bac92ce5-a40b-489b-9e35-7b8dc128fac7_isrForCrosstalkSources_1101474_11.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/bac92ce5-a40b-489b-9e35-7b8dc128fac7_isrForCrosstalkSources_1101474_11.stderr parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash de2b1ff46bc4634cee32a7b11934f6cd parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/5bb0144c-d0a0-46b7-9c40-986ea7381fff_isrForCrosstalkSources_1101465_36.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/5bb0144c-d0a0-46b7-9c40-986ea7381fff_isrForCrosstalkSources_1101465_36.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash d3a47f3e4fb6aa61817d1bfe29338fad parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-19 00:39:25 proc_lsst.multi:170 [INFO] [multi] job_id 17225951 proc_lsst.multi INFO: [multi] job_id 17225951 2024-03-19 00:39:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-19 00:39:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:25 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225951 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225951 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17225951 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-19 00:39:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-19 00:39:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-19 00:39:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/4167592d-0cdb-4921-88b4-84b7759a00b8_isrForCrosstalkSources_1101468_48.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/4167592d-0cdb-4921-88b4-84b7759a00b8_isrForCrosstalkSources_1101468_48.stderr parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash dc6a12a2109444cb5663430c295af36c parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/d8d4b7c7-6bb2-480c-8342-f33300392d3a_isrForCrosstalkSources_1101468_49.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/d8d4b7c7-6bb2-480c-8342-f33300392d3a_isrForCrosstalkSources_1101468_49.stderr parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash 87f74454f8749ba72e37483aea3692fc parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/e47d5e7f-5bd6-4bc4-860f-d55635cabd79_isrForCrosstalkSources_1101471_46.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/e47d5e7f-5bd6-4bc4-860f-d55635cabd79_isrForCrosstalkSources_1101471_46.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash b4e65b60fb4bae9b377aec17c6f9cb38 parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/c5634bf7-20c2-47a7-8257-eeffa4cabc6a_isrForCrosstalkSources_1101468_40.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/c5634bf7-20c2-47a7-8257-eeffa4cabc6a_isrForCrosstalkSources_1101468_40.stderr parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash bcd19f3798d3fbd35b3f1f963a355c2d parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498076480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/6733253b-8d2a-4e79-9f4c-e78a287653f2_isrForCrosstalkSources_1101465_11.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/6733253b-8d2a-4e79-9f4c-e78a287653f2_isrForCrosstalkSources_1101465_11.stderr parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash 9c223b57ffce0bd0c37cbed12bbabe38 parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1554980767a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/48bbf511-ac4a-4e59-9465-19407201ab00_isrForCrosstalkSources_1101468_11.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/48bbf511-ac4a-4e59-9465-19407201ab00_isrForCrosstalkSources_1101468_11.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash 34164c45187d897ce926126c34c36f9a parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498074040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/3a01ff1e-83f4-45f5-96c6-19dd99958624_isrForCrosstalkSources_1101468_20.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/3a01ff1e-83f4-45f5-96c6-19dd99958624_isrForCrosstalkSources_1101468_20.stderr parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash d10c4cfbad9dc8677767a23588f159fd parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498c1dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/794f8541-aa9a-47c6-bfeb-b0042dcda78b_isrForCrosstalkSources_1101465_2.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/794f8541-aa9a-47c6-bfeb-b0042dcda78b_isrForCrosstalkSources_1101465_2.stderr parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash bba11caa3d7d4ec736d45e37537049de parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498076520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/837afd65-172e-4849-91ae-fe288674cd78_isrForCrosstalkSources_1101468_29.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/837afd65-172e-4849-91ae-fe288674cd78_isrForCrosstalkSources_1101468_29.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 6b33f8eb81a114d179bb74e68418e60e parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498077920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/e5b1c622-0193-4d51-9b3f-1aa6ac09e521_isrForCrosstalkSources_1101474_35.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/e5b1c622-0193-4d51-9b3f-1aa6ac09e521_isrForCrosstalkSources_1101474_35.stderr parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash 61ae85e8d1204395b52090aaed33aa87 parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1554980742c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/0a92bcbc-bb4e-4f6d-85ba-395319182e7b_isrForCrosstalkSources_1101465_25.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/0a92bcbc-bb4e-4f6d-85ba-395319182e7b_isrForCrosstalkSources_1101465_25.stderr parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash dbf89e4d9863615c4e78ae63c2d4427f parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1554980779c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/aa33965d-da90-4669-af74-ee9bc8017873_isrForCrosstalkSources_1101465_9.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/aa33965d-da90-4669-af74-ee9bc8017873_isrForCrosstalkSources_1101465_9.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash b8b4a9fe35a97072f915003e7a59b8aa parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498077880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/f316a5e9-3090-4ab4-a155-fc599255d00c_isrForCrosstalkSources_1101471_51.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/f316a5e9-3090-4ab4-a155-fc599255d00c_isrForCrosstalkSources_1101471_51.stderr parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash afe7ad161c7ab7b0cbfb2cbefef43127 parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498077740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/4b86af2d-b7a9-4a19-870a-a9847d64af10_isrForCrosstalkSources_1101471_43.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/4b86af2d-b7a9-4a19-870a-a9847d64af10_isrForCrosstalkSources_1101471_43.stderr parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 2b4f62a75fbce0535061ededb6b00ead parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1554980740e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/f099f506-bfd5-4586-9f15-a2496b63857f_isrForCrosstalkSources_1101465_47.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/f099f506-bfd5-4586-9f15-a2496b63857f_isrForCrosstalkSources_1101465_47.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 6c163dd21325c36a0f8b04b3b9836713 parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1554980777e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/4a224eea-00f7-4771-b201-2c855579de51_isrForCrosstalkSources_1101474_33.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/4a224eea-00f7-4771-b201-2c855579de51_isrForCrosstalkSources_1101474_33.stderr parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash d0536a90340b21297061682fab87ba65 parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498074360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/38ec3768-d768-42c8-aacb-e59542af779e_isrForCrosstalkSources_1101471_42.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/38ec3768-d768-42c8-aacb-e59542af779e_isrForCrosstalkSources_1101471_42.stderr parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 0840a28b3ca5e1460f20f2c1dcaa511a parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498077600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/5ca15d87-8508-470a-9288-460e25ccb988_isrForCrosstalkSources_1101468_12.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/5ca15d87-8508-470a-9288-460e25ccb988_isrForCrosstalkSources_1101468_12.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 981b287b242e6e1593277e41c8564d72 parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498074540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/85f0a2b5-c86e-44d2-a636-abf8f849b140_isrForCrosstalkSources_1101471_19.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/85f0a2b5-c86e-44d2-a636-abf8f849b140_isrForCrosstalkSources_1101471_19.stderr parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 0a7bff1189eecb0a51182177f1662892 parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498074680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/585d8c72-ebde-41fa-9f73-76a7ee48ae0c_isrForCrosstalkSources_1101471_5.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/585d8c72-ebde-41fa-9f73-76a7ee48ae0c_isrForCrosstalkSources_1101471_5.stderr parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash e9f5d83aeaddc6b8d258111e4f6fb69c parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498074720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/4c17f1aa-eef4-4d8d-bc30-fc3b85432776_isrForCrosstalkSources_1101474_15.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/4c17f1aa-eef4-4d8d-bc30-fc3b85432776_isrForCrosstalkSources_1101474_15.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 581df70bb61afe4abd5eda55ec115240 parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1554980745e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/91042eb3-207b-4f06-964d-ed17484b5fa1_isrForCrosstalkSources_1101471_49.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/91042eb3-207b-4f06-964d-ed17484b5fa1_isrForCrosstalkSources_1101471_49.stderr parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash e1a3b01da470b40ae6f8e6cb1afa2696 parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1554980747c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/3f244fff-8b4a-4678-8121-1556d19f6429_isrForCrosstalkSources_1101474_52.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/3f244fff-8b4a-4678-8121-1556d19f6429_isrForCrosstalkSources_1101474_52.stderr parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash e3541c8a365619b8d3de629ea5d1e428 parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x155498074900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/19cf25d0-c3c5-4de0-8cfd-8def429126f4_isrForCrosstalkSources_1101468_17.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/19cf25d0-c3c5-4de0-8cfd-8def429126f4_isrForCrosstalkSources_1101468_17.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 69d2a98de29219137000d3cd369f39c1 parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1554980749a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/90281749-4ca9-41df-b04a-38b245eb86de_isrForCrosstalkSources_1101465_12.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/90281749-4ca9-41df-b04a-38b245eb86de_isrForCrosstalkSources_1101465_12.stderr 2024-03-19 00:39:25 proc_lsst.multi:170 [INFO] [multi] job_id 17225952 proc_lsst.multi INFO: [multi] job_id 17225952 2024-03-19 00:39:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-19 00:39:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:25 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225952 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225952 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17225952 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-19 00:39:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-19 00:39:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-19 00:39:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:25 proc_lsst.multi:170 [INFO] [multi] job_id 17225953 proc_lsst.multi INFO: [multi] job_id 17225953 2024-03-19 00:39:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-19 00:39:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:25 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225953 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225953 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17225953 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-19 00:39:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-19 00:39:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-19 00:39:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:25 proc_lsst.multi:170 [INFO] [multi] job_id 17225954 proc_lsst.multi INFO: [multi] job_id 17225954 2024-03-19 00:39:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:25 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225954 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225954 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17225954 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-19 00:39:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-19 00:39:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:25 proc_lsst.multi:170 [INFO] [multi] job_id 17225955 proc_lsst.multi INFO: [multi] job_id 17225955 2024-03-19 00:39:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-19 00:39:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:25 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225955 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225955 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17225955 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-19 00:39:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-19 00:39:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-19 00:39:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:26 proc_lsst.multi:170 [INFO] [multi] job_id 17225956 proc_lsst.multi INFO: [multi] job_id 17225956 2024-03-19 00:39:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-19 00:39:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:26 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225956 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225956 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17225956 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-19 00:39:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-19 00:39:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-19 00:39:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:26 proc_lsst.multi:170 [INFO] [multi] job_id 17225957 proc_lsst.multi INFO: [multi] job_id 17225957 2024-03-19 00:39:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-19 00:39:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:26 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225957 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225957 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17225957 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-19 00:39:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-19 00:39:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-19 00:39:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:26 proc_lsst.multi:170 [INFO] [multi] job_id 17225958 proc_lsst.multi INFO: [multi] job_id 17225958 2024-03-19 00:39:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-19 00:39:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:26 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225958 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225958 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17225958 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-19 00:39:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-19 00:39:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-19 00:39:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:26 proc_lsst.multi:170 [INFO] [multi] job_id 17225959 proc_lsst.multi INFO: [multi] job_id 17225959 2024-03-19 00:39:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-19 00:39:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:26 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225959 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225959 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17225959 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-19 00:39:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-19 00:39:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-19 00:39:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:27 proc_lsst.multi:170 [INFO] [multi] job_id 17225960 proc_lsst.multi INFO: [multi] job_id 17225960 2024-03-19 00:39:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-19 00:39:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:27 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225960 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225960 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17225960 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-19 00:39:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-19 00:39:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-19 00:39:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:27 proc_lsst.multi:170 [INFO] [multi] job_id 17225961 proc_lsst.multi INFO: [multi] job_id 17225961 2024-03-19 00:39:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-19 00:39:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:27 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225961 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225961 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17225961 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-19 00:39:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-19 00:39:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-19 00:39:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:27 proc_lsst.multi:170 [INFO] [multi] job_id 17225962 proc_lsst.multi INFO: [multi] job_id 17225962 2024-03-19 00:39:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-19 00:39:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:27 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225962 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225962 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17225962 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-19 00:39:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-19 00:39:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-19 00:39:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:28 proc_lsst.multi:170 [INFO] [multi] job_id 17225963 proc_lsst.multi INFO: [multi] job_id 17225963 2024-03-19 00:39:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-19 00:39:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:28 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225963 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225963 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17225963 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-19 00:39:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-19 00:39:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-19 00:39:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:29 proc_lsst.multi:170 [INFO] [multi] job_id 17225964 proc_lsst.multi INFO: [multi] job_id 17225964 2024-03-19 00:39:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-19 00:39:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:29 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225964 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225964 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17225964 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-19 00:39:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-19 00:39:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-19 00:39:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:29 proc_lsst.multi:170 [INFO] [multi] job_id 17225965 proc_lsst.multi INFO: [multi] job_id 17225965 2024-03-19 00:39:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-19 00:39:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:29 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225965 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225965 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17225965 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-19 00:39:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-19 00:39:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-19 00:39:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:29 proc_lsst.multi:170 [INFO] [multi] job_id 17225966 proc_lsst.multi INFO: [multi] job_id 17225966 2024-03-19 00:39:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-19 00:39:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:29 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225966 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225966 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17225966 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-19 00:39:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-19 00:39:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-19 00:39:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:30 proc_lsst.multi:170 [INFO] [multi] job_id 17225967 proc_lsst.multi INFO: [multi] job_id 17225967 2024-03-19 00:39:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-19 00:39:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:30 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225967 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225967 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17225967 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-19 00:39:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-19 00:39:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-19 00:39:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:30 proc_lsst.multi:170 [INFO] [multi] job_id 17225968 proc_lsst.multi INFO: [multi] job_id 17225968 2024-03-19 00:39:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-19 00:39:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:30 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225968 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225968 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17225968 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-19 00:39:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-19 00:39:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-19 00:39:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:30 proc_lsst.multi:170 [INFO] [multi] job_id 17225969 proc_lsst.multi INFO: [multi] job_id 17225969 2024-03-19 00:39:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-19 00:39:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:30 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225969 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225969 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17225969 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-19 00:39:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-19 00:39:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-19 00:39:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:31 proc_lsst.multi:170 [INFO] [multi] job_id 17225970 proc_lsst.multi INFO: [multi] job_id 17225970 2024-03-19 00:39:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-19 00:39:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:31 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225970 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225970 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17225970 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-19 00:39:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-19 00:39:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-19 00:39:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:31 proc_lsst.multi:170 [INFO] [multi] job_id 17225971 proc_lsst.multi INFO: [multi] job_id 17225971 2024-03-19 00:39:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-19 00:39:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:31 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225971 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225971 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17225971 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-19 00:39:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-19 00:39:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-19 00:39:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:32 proc_lsst.multi:170 [INFO] [multi] job_id 17225972 proc_lsst.multi INFO: [multi] job_id 17225972 2024-03-19 00:39:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-19 00:39:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:32 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225972 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225972 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17225972 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-19 00:39:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-19 00:39:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-19 00:39:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:32 proc_lsst.multi:170 [INFO] [multi] job_id 17225973 proc_lsst.multi INFO: [multi] job_id 17225973 2024-03-19 00:39:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-19 00:39:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:32 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225973 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225973 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17225973 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-19 00:39:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-19 00:39:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-19 00:39:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:33 proc_lsst.multi:170 [INFO] [multi] job_id 17225974 proc_lsst.multi INFO: [multi] job_id 17225974 2024-03-19 00:39:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-19 00:39:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:33 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225974 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225974 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17225974 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-19 00:39:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-19 00:39:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-19 00:39:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:33 proc_lsst.multi:170 [INFO] [multi] job_id 17225975 proc_lsst.multi INFO: [multi] job_id 17225975 2024-03-19 00:39:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-19 00:39:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:33 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225975 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225975 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17225975 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-19 00:39:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-19 00:39:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-19 00:39:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:34 proc_lsst.multi:170 [INFO] [multi] job_id 17225976 proc_lsst.multi INFO: [multi] job_id 17225976 2024-03-19 00:39:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-19 00:39:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:34 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225976 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225976 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17225976 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-19 00:39:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-19 00:39:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-19 00:39:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:34 proc_lsst.multi:170 [INFO] [multi] job_id 17225977 proc_lsst.multi INFO: [multi] job_id 17225977 2024-03-19 00:39:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-19 00:39:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:34 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225977 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225977 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17225977 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-19 00:39:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-19 00:39:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-19 00:39:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:35 proc_lsst.multi:170 [INFO] [multi] job_id 17225978 proc_lsst.multi INFO: [multi] job_id 17225978 2024-03-19 00:39:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-19 00:39:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:35 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225978 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225978 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17225978 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-19 00:39:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-19 00:39:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-19 00:39:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:35 proc_lsst.multi:170 [INFO] [multi] job_id 17225979 proc_lsst.multi INFO: [multi] job_id 17225979 2024-03-19 00:39:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:35 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17225979 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17225979 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17225979 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-19 00:39:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-19 00:39:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-19 00:39:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:37 proc_lsst.multi:170 [INFO] [multi] job_id 17225980 proc_lsst.multi INFO: [multi] job_id 17225980 2024-03-19 00:39:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:37 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17225980 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17225980 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17225980 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-19 00:39:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-19 00:39:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:38 proc_lsst.multi:170 [INFO] [multi] job_id 17225981 proc_lsst.multi INFO: [multi] job_id 17225981 2024-03-19 00:39:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-19 00:39:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:38 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17225981 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17225981 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17225981 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-19 00:39:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-19 00:39:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-19 00:39:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 187b2207b41f584570e9524a7126ccca with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/f34031eb-b745-4710-aace-fa4857ebaf3e_isrForCrosstalkSources_1101474_62.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/f34031eb-b745-4710-aace-fa4857ebaf3e_isrForCrosstalkSources_1101474_62.stderr 2024-03-19 00:39:40 proc_lsst.multi:170 [INFO] [multi] job_id 17225982 proc_lsst.multi INFO: [multi] job_id 17225982 2024-03-19 00:39:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-19 00:39:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:40 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17225982 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17225982 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17225982 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-19 00:39:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-19 00:39:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-19 00:39:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 401f46446f8e1eb11d5c6cf8e53e52f9 with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/a2c71ca4-7f0a-486f-9623-512ca6ff9e5c_isrForCrosstalkSources_1101468_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/a2c71ca4-7f0a-486f-9623-512ca6ff9e5c_isrForCrosstalkSources_1101468_28.stderr 2024-03-19 00:39:41 proc_lsst.multi:170 [INFO] [multi] job_id 17225983 proc_lsst.multi INFO: [multi] job_id 17225983 2024-03-19 00:39:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-19 00:39:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:41 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17225983 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17225983 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17225983 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-19 00:39:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-19 00:39:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-19 00:39:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:42 proc_lsst.multi:170 [INFO] [multi] job_id 17225984 proc_lsst.multi INFO: [multi] job_id 17225984 2024-03-19 00:39:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:42 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17225984 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17225984 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17225984 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-19 00:39:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-19 00:39:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-19 00:39:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:43 proc_lsst.multi:170 [INFO] [multi] job_id 17225985 proc_lsst.multi INFO: [multi] job_id 17225985 2024-03-19 00:39:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:43 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17225985 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17225985 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17225985 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-19 00:39:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-19 00:39:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:44 proc_lsst.multi:170 [INFO] [multi] job_id 17225986 proc_lsst.multi INFO: [multi] job_id 17225986 2024-03-19 00:39:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-19 00:39:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:44 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17225986 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17225986 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17225986 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-19 00:39:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-19 00:39:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-19 00:39:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:45 proc_lsst.multi:170 [INFO] [multi] job_id 17225987 proc_lsst.multi INFO: [multi] job_id 17225987 2024-03-19 00:39:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-19 00:39:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:45 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17225987 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17225987 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17225987 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-19 00:39:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-19 00:39:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-19 00:39:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:45 proc_lsst.multi:170 [INFO] [multi] job_id 17225988 proc_lsst.multi INFO: [multi] job_id 17225988 2024-03-19 00:39:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-19 00:39:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:45 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17225988 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17225988 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17225988 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-19 00:39:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-19 00:39:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-19 00:39:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:46 proc_lsst.multi:170 [INFO] [multi] job_id 17225989 proc_lsst.multi INFO: [multi] job_id 17225989 2024-03-19 00:39:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:46 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17225989 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17225989 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17225989 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-19 00:39:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-19 00:39:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:48 proc_lsst.multi:170 [INFO] [multi] job_id 17225990 proc_lsst.multi INFO: [multi] job_id 17225990 2024-03-19 00:39:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-19 00:39:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:48 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17225990 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17225990 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17225990 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23453085687824 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 42, active_tasks = 246 parsl.dataflow.strategy DEBUG: Executor multi has 246 active tasks, 1/41 running/pending blocks, and 30 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 42 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 20 more blocks parsl.executors.status_handling INFO: Scaling out by 20 blocks parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-19 00:39:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-19 00:39:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-19 00:39:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd0cf8a26f1583b5cedfc89c9e4ac9bd with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/3d573f99-e69e-4188-8510-bbc807af21c0_isrForCrosstalkSources_1101468_8.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/3d573f99-e69e-4188-8510-bbc807af21c0_isrForCrosstalkSources_1101468_8.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9cbaef72fd3f7334579355d731122d2 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/ada1bee2-b2ad-45a7-a7ac-81846d5d20ac_isrForCrosstalkSources_1101474_26.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/ada1bee2-b2ad-45a7-a7ac-81846d5d20ac_isrForCrosstalkSources_1101474_26.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ea7bae8134bed0da84db7eb7cc963d3 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/8f3e22ea-1058-4ed8-9958-7fc76886df29_isrForCrosstalkSources_1101474_45.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/8f3e22ea-1058-4ed8-9958-7fc76886df29_isrForCrosstalkSources_1101474_45.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fe5b352644fc8369d81d53a28a5b25a with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/079368a0-64dc-4819-a6ce-804316ecfb41_isrForCrosstalkSources_1101474_20.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/079368a0-64dc-4819-a6ce-804316ecfb41_isrForCrosstalkSources_1101474_20.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34caeaec523ab66c2754231aacbd82c0 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/b65b1bbd-17da-40da-8f08-68eec6af7717_isrForCrosstalkSources_1101474_56.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/b65b1bbd-17da-40da-8f08-68eec6af7717_isrForCrosstalkSources_1101474_56.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c8943d959ad15bf803c7a4b10d32978 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/96f47baa-aa21-4e03-ad9c-2c09beea2ff4_isrForCrosstalkSources_1101471_48.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/96f47baa-aa21-4e03-ad9c-2c09beea2ff4_isrForCrosstalkSources_1101471_48.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3645338f1d9360d733051d99769d99b8 with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/08d80e02-45cf-4cca-8249-7781459129d4_isrForCrosstalkSources_1101471_14.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/08d80e02-45cf-4cca-8249-7781459129d4_isrForCrosstalkSources_1101471_14.stderr 2024-03-19 00:39:49 proc_lsst.multi:170 [INFO] [multi] job_id 17225991 proc_lsst.multi INFO: [multi] job_id 17225991 2024-03-19 00:39:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-19 00:39:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:49 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17225991 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17225991 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17225991 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-19 00:39:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-19 00:39:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-19 00:39:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4579096a32c9f2155abc0898b87b8989 with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/8a2a4216-09ec-4a0d-b6f2-6f6ebafc8d97_isrForCrosstalkSources_1101471_17.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/8a2a4216-09ec-4a0d-b6f2-6f6ebafc8d97_isrForCrosstalkSources_1101471_17.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bf8984b1a802f881b07398e5855452d with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/75ab27aa-888e-402a-b904-c90c4bc6b0a4_isrForCrosstalkSources_1101471_58.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/75ab27aa-888e-402a-b904-c90c4bc6b0a4_isrForCrosstalkSources_1101471_58.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e01c129e88b09351e325eba20d8da69 with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/91f19c82-79bf-4a9f-b49f-444f5f4d4309_isrForCrosstalkSources_1101465_39.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/91f19c82-79bf-4a9f-b49f-444f5f4d4309_isrForCrosstalkSources_1101465_39.stderr 2024-03-19 00:39:50 proc_lsst.multi:170 [INFO] [multi] job_id 17225992 proc_lsst.multi INFO: [multi] job_id 17225992 2024-03-19 00:39:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-19 00:39:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:50 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17225992 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17225992 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17225992 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-19 00:39:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-19 00:39:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-19 00:39:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a09b168ce6a073b98ea89be92acde44 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/21e7102d-1330-4485-9db0-a7e7c9775603_isrForCrosstalkSources_1101474_38.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/21e7102d-1330-4485-9db0-a7e7c9775603_isrForCrosstalkSources_1101474_38.stderr parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:51 proc_lsst.multi:170 [INFO] [multi] job_id 17225993 proc_lsst.multi INFO: [multi] job_id 17225993 2024-03-19 00:39:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-19 00:39:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:51 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17225993 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17225993 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17225993 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-19 00:39:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-19 00:39:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-19 00:39:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:51 proc_lsst.multi:170 [INFO] [multi] job_id 17225994 proc_lsst.multi INFO: [multi] job_id 17225994 2024-03-19 00:39:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-19 00:39:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:51 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17225994 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17225994 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17225994 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-19 00:39:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-19 00:39:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-19 00:39:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af874b35edd16f63b836b17c4fbda0d0 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/1dad9f89-3f4c-420b-83cd-c61d13cef7ba_isrForCrosstalkSources_1101465_1.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/1dad9f89-3f4c-420b-83cd-c61d13cef7ba_isrForCrosstalkSources_1101465_1.stderr 2024-03-19 00:39:52 proc_lsst.multi:170 [INFO] [multi] job_id 17225995 proc_lsst.multi INFO: [multi] job_id 17225995 2024-03-19 00:39:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-19 00:39:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:52 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17225995 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17225995 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 17225995 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-19 00:39:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-19 00:39:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-19 00:39:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01fc1dc6a02b5ecea9ab0615d3b16520 with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/90db54a6-590f-47e2-9eaf-5d1db4da62e7_isrForCrosstalkSources_1101465_30.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/90db54a6-590f-47e2-9eaf-5d1db4da62e7_isrForCrosstalkSources_1101465_30.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ad61a18f290e2ee1d301668523c89f6 with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/0443233d-e633-419d-bfb4-f39eadc00931_isrForCrosstalkSources_1101465_41.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/0443233d-e633-419d-bfb4-f39eadc00931_isrForCrosstalkSources_1101465_41.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecc2f3cd11a6058bc67602d019b5094e with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/58c43dfa-25ba-4e98-bfaf-d602e12b734d_isrForCrosstalkSources_1101468_53.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/58c43dfa-25ba-4e98-bfaf-d602e12b734d_isrForCrosstalkSources_1101468_53.stderr 2024-03-19 00:39:52 proc_lsst.multi:170 [INFO] [multi] job_id 17225996 proc_lsst.multi INFO: [multi] job_id 17225996 2024-03-19 00:39:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-19 00:39:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:52 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17225996 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17225996 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 17225996 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-19 00:39:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-19 00:39:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-19 00:39:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c57ece3be8807998ad6b2638121e7ffc with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/7690426e-d29a-42fd-891d-7562a610f3f7_isrForCrosstalkSources_1101465_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/7690426e-d29a-42fd-891d-7562a610f3f7_isrForCrosstalkSources_1101465_33.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddc4c128e66255f8445febcb96a8dc7f with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/ea33079e-7f82-4c06-b8e8-e2bf6911d196_isrForCrosstalkSources_1101471_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/ea33079e-7f82-4c06-b8e8-e2bf6911d196_isrForCrosstalkSources_1101471_45.stderr parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bec4a4600289c4a46095f96d59c8d8b5 with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/9bc00ad6-7e62-496b-920c-fd7a79299682_isrForCrosstalkSources_1101471_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/9bc00ad6-7e62-496b-920c-fd7a79299682_isrForCrosstalkSources_1101471_15.stderr 2024-03-19 00:39:54 proc_lsst.multi:170 [INFO] [multi] job_id 17225997 proc_lsst.multi INFO: [multi] job_id 17225997 2024-03-19 00:39:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-19 00:39:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:54 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17225997 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17225997 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 17225997 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-19 00:39:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-19 00:39:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-19 00:39:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50edf6b5cabbb72c4c82116af55d8895 with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/03ef263c-ec1a-45ad-b3ef-b62b116c58da_isrForCrosstalkSources_1101474_9.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/03ef263c-ec1a-45ad-b3ef-b62b116c58da_isrForCrosstalkSources_1101474_9.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d65d317806d8bed9a87720d75e1d2fa with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/3fcdb653-8fec-4418-b3a4-07fe78650f30_isrForCrosstalkSources_1101465_21.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/3fcdb653-8fec-4418-b3a4-07fe78650f30_isrForCrosstalkSources_1101465_21.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bd3b1844a244ad8246083d41746d753 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/c6ebf8d4-fdae-4abb-95a7-437a0bf19c00_isrForCrosstalkSources_1101468_51.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/c6ebf8d4-fdae-4abb-95a7-437a0bf19c00_isrForCrosstalkSources_1101468_51.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22ef6819bc6b40a866cda31d63719ae2 with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/f65cae74-693b-4883-9f31-d3d127210deb_isrForCrosstalkSources_1101471_7.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/f65cae74-693b-4883-9f31-d3d127210deb_isrForCrosstalkSources_1101471_7.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e15c86ecfa7825893ca5bf4261d2f03b with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/4fad8c51-51c3-47c0-9d36-64a650ddc82d_isrForCrosstalkSources_1101474_58.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/4fad8c51-51c3-47c0-9d36-64a650ddc82d_isrForCrosstalkSources_1101474_58.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d269a0f5cf38f0e4a32218a78061a074 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/2514b612-4df2-4884-bbde-0a264749760b_isrForCrosstalkSources_1101471_9.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/2514b612-4df2-4884-bbde-0a264749760b_isrForCrosstalkSources_1101471_9.stderr 2024-03-19 00:39:55 proc_lsst.multi:170 [INFO] [multi] job_id 17225998 proc_lsst.multi INFO: [multi] job_id 17225998 2024-03-19 00:39:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-19 00:39:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:55 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17225998 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17225998 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 17225998 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-19 00:39:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-19 00:39:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-19 00:39:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:56 proc_lsst.multi:170 [INFO] [multi] job_id 17225999 proc_lsst.multi INFO: [multi] job_id 17225999 2024-03-19 00:39:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-19 00:39:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:56 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17225999 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17225999 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 17225999 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-19 00:39:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-19 00:39:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-19 00:39:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:57 proc_lsst.multi:170 [INFO] [multi] job_id 17226000 proc_lsst.multi INFO: [multi] job_id 17226000 2024-03-19 00:39:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-19 00:39:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:57 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226000 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226000 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 17226000 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-19 00:39:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-19 00:39:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-19 00:39:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08c848e45a8c9abfc2d81e2e06999488 with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/452ba558-1c65-4489-a013-10f154a695e4_isrForCrosstalkSources_1101465_58.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/452ba558-1c65-4489-a013-10f154a695e4_isrForCrosstalkSources_1101465_58.stderr 2024-03-19 00:39:57 proc_lsst.multi:170 [INFO] [multi] job_id 17226001 proc_lsst.multi INFO: [multi] job_id 17226001 2024-03-19 00:39:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-19 00:39:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:57 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226001 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226001 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 17226001 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-19 00:39:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-19 00:39:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-19 00:39:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:39:59 proc_lsst.multi:170 [INFO] [multi] job_id 17226002 proc_lsst.multi INFO: [multi] job_id 17226002 2024-03-19 00:39:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-19 00:39:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:59 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226002 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226002 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 17226002 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-19 00:39:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-19 00:39:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:39:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:39:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:39:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:39:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:39:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:39:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:39:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:39:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-19 00:39:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:39:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b926f259150611a59dabe97d779adf63 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/d4f919d9-12df-432d-bd25-f8a9f587ccb1_isrForCrosstalkSources_1101465_38.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/d4f919d9-12df-432d-bd25-f8a9f587ccb1_isrForCrosstalkSources_1101465_38.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b86a8742115c30cdc6ee98b28001d7f with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/800eb2de-e042-4ed3-8ccb-14287b43a7b1_isrForCrosstalkSources_1101465_7.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/800eb2de-e042-4ed3-8ccb-14287b43a7b1_isrForCrosstalkSources_1101465_7.stderr parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 362f67f9ad95c3afb163ae96bcead803 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/4c330de1-9f83-467e-a472-3003f9d17445_isrForCrosstalkSources_1101468_22.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/4c330de1-9f83-467e-a472-3003f9d17445_isrForCrosstalkSources_1101468_22.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bef2054d013be25abf03fbd4ee0abd2f with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/a8ee3cec-cdaa-4cb2-846d-0f9643cc54af_isrForCrosstalkSources_1101465_43.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/a8ee3cec-cdaa-4cb2-846d-0f9643cc54af_isrForCrosstalkSources_1101465_43.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4d54e80703572fe8ac4b857741a5a02 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/481db7a6-f89b-4f7c-8d1b-b035996c2253_isrForCrosstalkSources_1101474_32.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/481db7a6-f89b-4f7c-8d1b-b035996c2253_isrForCrosstalkSources_1101474_32.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9752593830886e12f8af795df6febbd8 with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/b747d873-d0c7-4ee0-b41f-f06ec1aa2919_isrForCrosstalkSources_1101471_31.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/b747d873-d0c7-4ee0-b41f-f06ec1aa2919_isrForCrosstalkSources_1101471_31.stderr parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50686e686d1cfb4ad9db8ae3dc5934b5 with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/f7af661a-826b-465d-ba3a-e4d38b008f33_isrForCrosstalkSources_1101471_41.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/f7af661a-826b-465d-ba3a-e4d38b008f33_isrForCrosstalkSources_1101471_41.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0dc107515f2c0dfe1428cb9bec66818a with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/f5db92f8-5b63-46fc-8376-2696b496e84e_isrForCrosstalkSources_1101474_6.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/f5db92f8-5b63-46fc-8376-2696b496e84e_isrForCrosstalkSources_1101474_6.stderr 2024-03-19 00:40:01 proc_lsst.multi:170 [INFO] [multi] job_id 17226003 proc_lsst.multi INFO: [multi] job_id 17226003 2024-03-19 00:40:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-19 00:40:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:40:01 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226003 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226003 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 17226003 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-19 00:40:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-19 00:40:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:40:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:40:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:40:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:40:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:40:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:40:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:40:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-19 00:40:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:40:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 752959034ccfe06ab252234c50a4f1db with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/5380299a-97ca-4c39-8aac-04266325188e_isrForCrosstalkSources_1101468_45.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/5380299a-97ca-4c39-8aac-04266325188e_isrForCrosstalkSources_1101468_45.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2df286b16a69736692cdfc315b2c0557 with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/2b3f00f9-9e0e-4f8d-a2a3-2600c7d869e0_isrForCrosstalkSources_1101471_38.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/2b3f00f9-9e0e-4f8d-a2a3-2600c7d869e0_isrForCrosstalkSources_1101471_38.stderr parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b99f5e3220bc4b30721832a94737ed17 with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/47432b96-ca72-4876-ab00-358ddac5baa7_isrForCrosstalkSources_1101468_27.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/47432b96-ca72-4876-ab00-358ddac5baa7_isrForCrosstalkSources_1101468_27.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 102c2484e36db8064f2947c2cce70c0c with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/eedbead3-011d-474e-8e4b-1eb7e9d4b1bf_isrForCrosstalkSources_1101474_22.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/eedbead3-011d-474e-8e4b-1eb7e9d4b1bf_isrForCrosstalkSources_1101474_22.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 833577b26507c83b48f455066d71dba4 with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/0bd6c4af-5e3e-454a-b2b8-85fc855e981e_isrForCrosstalkSources_1101474_18.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/0bd6c4af-5e3e-454a-b2b8-85fc855e981e_isrForCrosstalkSources_1101474_18.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce6234e9e839a1bf8294276e1dac4e77 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/c4d0b8ac-541d-4b72-b52c-e984f10f40af_isrForCrosstalkSources_1101474_39.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/c4d0b8ac-541d-4b72-b52c-e984f10f40af_isrForCrosstalkSources_1101474_39.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f772136bd2e20fa8b9d61b67bb27489 with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/dac31026-f676-4ec3-96b4-850dcc2bb1fd_isrForCrosstalkSources_1101468_55.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/dac31026-f676-4ec3-96b4-850dcc2bb1fd_isrForCrosstalkSources_1101468_55.stderr parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da0a581431fef853aabd9a98d99b3a60 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/722d568f-b6e7-4ff6-95fb-e2421ce094a3_isrForCrosstalkSources_1101474_28.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/722d568f-b6e7-4ff6-95fb-e2421ce094a3_isrForCrosstalkSources_1101474_28.stderr 2024-03-19 00:40:04 proc_lsst.multi:170 [INFO] [multi] job_id 17226004 proc_lsst.multi INFO: [multi] job_id 17226004 2024-03-19 00:40:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-19 00:40:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:40:04 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17226004 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17226004 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 17226004 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-19 00:40:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-19 00:40:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:40:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:40:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:40:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:40:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:40:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:40:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:40:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-19 00:40:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:40:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-19 00:40:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 00:40:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd5b65db4f79925c223ae32e40a498c4 with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/3fd32e61-2876-4daf-9243-880b8f11f6a3_isrForCrosstalkSources_1101465_22.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/3fd32e61-2876-4daf-9243-880b8f11f6a3_isrForCrosstalkSources_1101465_22.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df00fcc848cc2da0be585a90daa25ba3 with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/da9a2325-ff7f-4a71-a9df-dd5b766ee272_isrForCrosstalkSources_1101468_21.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/da9a2325-ff7f-4a71-a9df-dd5b766ee272_isrForCrosstalkSources_1101468_21.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bb86d794670500444c2293f051d7773 with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/5b04112e-7777-49c3-a791-fad4b0097801_isrForCrosstalkSources_1101474_24.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/5b04112e-7777-49c3-a791-fad4b0097801_isrForCrosstalkSources_1101474_24.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03e7f0287476af7ab5bce4cc088f8681 with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/53d19e00-ed9a-41b2-a2ef-19ac81346082_isrForCrosstalkSources_1101465_51.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/53d19e00-ed9a-41b2-a2ef-19ac81346082_isrForCrosstalkSources_1101465_51.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8e49dbe01320cd5d820fc964fdea43c with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/f59944fb-c5e2-46ee-9570-41183756ec81_isrForCrosstalkSources_1101468_58.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/f59944fb-c5e2-46ee-9570-41183756ec81_isrForCrosstalkSources_1101468_58.stderr parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 651bfdbdf529724bc77f3186b25d48c7 with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/557f7149-24d5-44d8-94af-a68f1fee19a5_isrForCrosstalkSources_1101471_12.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/557f7149-24d5-44d8-94af-a68f1fee19a5_isrForCrosstalkSources_1101471_12.stderr parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e26077297a4b09848ea047e0125ae4fc with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/7edda041-fc9f-4597-9a03-2caa813b4b93_isrForCrosstalkSources_1101468_59.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/7edda041-fc9f-4597-9a03-2caa813b4b93_isrForCrosstalkSources_1101468_59.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 673d3a1de488963756275dfc625bd032 with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/aa7c637e-0e90-4233-a780-77eba01822c8_isrForCrosstalkSources_1101474_4.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/aa7c637e-0e90-4233-a780-77eba01822c8_isrForCrosstalkSources_1101474_4.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 829df29b2e8eee3d90a6fa2b7c145313 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/21a862b3-d64e-4f6a-8e61-817d637f2f27_isrForCrosstalkSources_1101471_4.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/21a862b3-d64e-4f6a-8e61-817d637f2f27_isrForCrosstalkSources_1101471_4.stderr parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9da27a08c7ab147ab8e43fed7ca0076 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/3a5ffb9e-aa90-4166-a3c8-41bdb2b7b74e_isrForCrosstalkSources_1101471_10.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/3a5ffb9e-aa90-4166-a3c8-41bdb2b7b74e_isrForCrosstalkSources_1101471_10.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2563125fe427b4b1f04bf159dadc8fb5 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/4aad3dd3-d6a6-4094-8c97-e72f7bd2c1d8_isrForCrosstalkSources_1101465_15.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/4aad3dd3-d6a6-4094-8c97-e72f7bd2c1d8_isrForCrosstalkSources_1101465_15.stderr 2024-03-19 00:40:07 proc_lsst.multi:170 [INFO] [multi] job_id 17226005 proc_lsst.multi INFO: [multi] job_id 17226005 2024-03-19 00:40:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:40:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 00:40:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17226005 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17226005 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 17226005 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-19 00:40:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-19 00:40:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:40:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:40:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:40:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:40:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:40:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:40:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:40:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-19 00:40:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:40:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:40:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 00:40:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 369285c0552012a65be31beb979bc238 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/cf3502a7-68e9-46ac-b93e-9dfc41b64cf0_isrForCrosstalkSources_1101474_50.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/cf3502a7-68e9-46ac-b93e-9dfc41b64cf0_isrForCrosstalkSources_1101474_50.stderr parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1288d5acf7f398523a1fd64cb9a7dda4 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/bb63d8d8-a161-40cd-8912-97f0cdd6ebf4_isrForCrosstalkSources_1101474_61.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/bb63d8d8-a161-40cd-8912-97f0cdd6ebf4_isrForCrosstalkSources_1101474_61.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f76dd45a75ab8d3f2d116e687816a087 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/97dda7cf-5aae-4737-a3e6-1dc12bef8cc0_isrForCrosstalkSources_1101474_36.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/97dda7cf-5aae-4737-a3e6-1dc12bef8cc0_isrForCrosstalkSources_1101474_36.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89b0254a0309a33800f9a12ba35a99ed with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/ce41e099-f74e-4d21-88b0-80ff95faf956_isrForCrosstalkSources_1101465_3.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/ce41e099-f74e-4d21-88b0-80ff95faf956_isrForCrosstalkSources_1101465_3.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0da2834dafeb17510e30e8900f64584 with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/47ea04d7-4b65-4d8a-b19b-419f6f428a24_isrForCrosstalkSources_1101465_28.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/47ea04d7-4b65-4d8a-b19b-419f6f428a24_isrForCrosstalkSources_1101465_28.stderr 2024-03-19 00:40:09 proc_lsst.multi:170 [INFO] [multi] job_id 17226006 proc_lsst.multi INFO: [multi] job_id 17226006 2024-03-19 00:40:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-19 00:40:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 00:40:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17226006 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17226006 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 17226006 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-19 00:40:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-19 00:40:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:40:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:40:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:40:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:40:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:40:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:40:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:40:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-19 00:40:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:40:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-19 00:40:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 00:40:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:40:11 proc_lsst.multi:170 [INFO] [multi] job_id 17226007 proc_lsst.multi INFO: [multi] job_id 17226007 2024-03-19 00:40:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-19 00:40:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 00:40:11 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17226007 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17226007 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 17226007 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-19 00:40:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-19 00:40:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:40:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:40:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:40:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:40:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:40:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:40:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:40:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-19 00:40:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:40:11 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-19 00:40:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 00:40:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-19 00:40:12 proc_lsst.multi:170 [INFO] [multi] job_id 17226008 proc_lsst.multi INFO: [multi] job_id 17226008 2024-03-19 00:40:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-19 00:40:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 00:40:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17226008 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17226008 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 17226008 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-19 00:40:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-19 00:40:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:40:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:40:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:40:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:40:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:40:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:40:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:40:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-19 00:40:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:40:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-19 00:40:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 00:40:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dcf6f083a7c10440b77294cb788460c with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/a4c97ae0-f7d1-4b7d-8622-80cd6fd6c49c_isrForCrosstalkSources_1101465_5.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/a4c97ae0-f7d1-4b7d-8622-80cd6fd6c49c_isrForCrosstalkSources_1101465_5.stderr parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ca932b389e86079ac63c7ad2dec358b with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/9002041b-11f9-4c07-83a9-9774c0cb5a46_isrForCrosstalkSources_1101474_48.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/9002041b-11f9-4c07-83a9-9774c0cb5a46_isrForCrosstalkSources_1101474_48.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21f5675624f290ed0410ad803637cbb1 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/9f466496-948e-4ac8-bd11-6e83fa9a3451_isrForCrosstalkSources_1101474_49.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/9f466496-948e-4ac8-bd11-6e83fa9a3451_isrForCrosstalkSources_1101474_49.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37dbf83c610d904b752377087635c889 with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/20d84bba-d561-4871-bc24-828bc5ebef2e_isrForCrosstalkSources_1101474_21.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/20d84bba-d561-4871-bc24-828bc5ebef2e_isrForCrosstalkSources_1101474_21.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ac4b0482f13e80f8b256852fbba7259 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/7d8ee1a7-4422-4598-b87e-aa64bea15bdc_isrForCrosstalkSources_1101468_32.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/7d8ee1a7-4422-4598-b87e-aa64bea15bdc_isrForCrosstalkSources_1101468_32.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de8ff57ec49e9e052d79f3b064a50f26 with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/a528da25-034e-4e90-8171-0f3c1494660f_isrForCrosstalkSources_1101468_34.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/a528da25-034e-4e90-8171-0f3c1494660f_isrForCrosstalkSources_1101468_34.stderr 2024-03-19 00:40:13 proc_lsst.multi:170 [INFO] [multi] job_id 17226009 proc_lsst.multi INFO: [multi] job_id 17226009 2024-03-19 00:40:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:40:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 00:40:13 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17226009 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17226009 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 17226009 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-19 00:40:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-19 00:40:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-19 00:40:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-19 00:40:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-19 00:40:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-19 00:40:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-19 00:40:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:40:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-19 00:40:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-19 00:40:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-19 00:40:13 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-19 00:40:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-19 00:40:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 00:40:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,127.0.0.1,10.64.129.9,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54962 --result_port=54176 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5686e11d665219940fbcd69982544851 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/225d0511-3083-442c-824c-5d420b42b755_isrForCrosstalkSources_1101465_46.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/225d0511-3083-442c-824c-5d420b42b755_isrForCrosstalkSources_1101465_46.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1f3e9525adc073e657de650f74efeed with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/65f09ad8-5a08-4fa2-8258-aed131b483a7_isrForCrosstalkSources_1101465_4.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/65f09ad8-5a08-4fa2-8258-aed131b483a7_isrForCrosstalkSources_1101465_4.stderr parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8c72b849bd0feb2ef1af75a8c11f585 with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/38fcfa64-c923-4d69-9870-a789e986e74b_isrForCrosstalkSources_1101465_50.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/38fcfa64-c923-4d69-9870-a789e986e74b_isrForCrosstalkSources_1101465_50.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b6cac546084bf60e8d02e430e7ce0a4 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/b7e3a08b-9057-4b3e-a407-eda893c088a0_isrForCrosstalkSources_1101468_23.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/b7e3a08b-9057-4b3e-a407-eda893c088a0_isrForCrosstalkSources_1101468_23.stderr parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63d94c0617f11eb3c6b6ced6f998c028 with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/01ace463-65d9-47f3-8d39-3f212bdcb723_isrForCrosstalkSources_1101465_42.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/01ace463-65d9-47f3-8d39-3f212bdcb723_isrForCrosstalkSources_1101465_42.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e39c8931fbc9e46a4475960f12fd4915 with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/00b92379-145f-42d5-85d1-eb7fbaef31ee_isrForCrosstalkSources_1101468_37.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/00b92379-145f-42d5-85d1-eb7fbaef31ee_isrForCrosstalkSources_1101468_37.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c7a0721243480970bbbc2b1b514e495 with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/12fca3f4-d2b2-424d-895b-f1922a63c00e_isrForCrosstalkSources_1101474_43.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/12fca3f4-d2b2-424d-895b-f1922a63c00e_isrForCrosstalkSources_1101474_43.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e2d7f59b0e51c01e3fadf2c6cad5619 with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/60ea1821-b012-486b-ad88-26dd73709904_isrForCrosstalkSources_1101471_2.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/60ea1821-b012-486b-ad88-26dd73709904_isrForCrosstalkSources_1101471_2.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52d00d6e98e7a9479499c635f9ecec89 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/6575c630-ddee-4ccd-ac26-4dfbcd0ca1f2_isrForCrosstalkSources_1101471_18.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/6575c630-ddee-4ccd-ac26-4dfbcd0ca1f2_isrForCrosstalkSources_1101471_18.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 312e607aae331c00f12bf99ffb253966 with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/cc48e68c-be54-4a26-8521-fc6c53e0053e_isrForCrosstalkSources_1101474_51.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/cc48e68c-be54-4a26-8521-fc6c53e0053e_isrForCrosstalkSources_1101474_51.stderr parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26594b3c1af3094b55b2d4238682c45b with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/3086e792-ff3b-42ef-86b1-4dc27fbd811a_isrForCrosstalkSources_1101471_3.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/3086e792-ff3b-42ef-86b1-4dc27fbd811a_isrForCrosstalkSources_1101471_3.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b2372fff7319e1e43f95306fd403235 with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/bc0d1640-4e3e-49ae-b398-d03c93f15cdc_isrForCrosstalkSources_1101474_55.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/bc0d1640-4e3e-49ae-b398-d03c93f15cdc_isrForCrosstalkSources_1101474_55.stderr parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72708528e9502138aa7932023c82d2ea with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/04b4c9fa-6db2-4cca-bdb5-81d2ee8c9e2a_isrForCrosstalkSources_1101465_19.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/04b4c9fa-6db2-4cca-bdb5-81d2ee8c9e2a_isrForCrosstalkSources_1101465_19.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e0003b08b800f5ecdd3405cb790dbf7 with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/6283eb8e-06f3-4b19-895a-963b2a09f3bc_isrForCrosstalkSources_1101468_14.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/6283eb8e-06f3-4b19-895a-963b2a09f3bc_isrForCrosstalkSources_1101468_14.stderr parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3146c224b555546f4fa85349284693b with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/2fe0418a-fb2f-4f5f-85cd-ceead2a2f193_isrForCrosstalkSources_1101468_4.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/2fe0418a-fb2f-4f5f-85cd-ceead2a2f193_isrForCrosstalkSources_1101468_4.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd521218a216ca394ed4a6368896ae81 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/036154ca-d642-4c98-a577-72ae525d3601_isrForCrosstalkSources_1101468_15.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/036154ca-d642-4c98-a577-72ae525d3601_isrForCrosstalkSources_1101468_15.stderr parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbd31f20c3a12223f7bfe36eda1d13df with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/68a6ecac-fffd-4790-b3cd-aa7ac87ac010_isrForCrosstalkSources_1101465_49.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/68a6ecac-fffd-4790-b3cd-aa7ac87ac010_isrForCrosstalkSources_1101465_49.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f5d06c3245ce575e379ed1c225492ae with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/2cf08127-5275-45be-9512-80d1ac559cc2_isrForCrosstalkSources_1101465_57.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/2cf08127-5275-45be-9512-80d1ac559cc2_isrForCrosstalkSources_1101465_57.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56796f390487e1567c1b6f674f1b5a54 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/99095763-3998-49bb-bdc9-60307b3414ae_isrForCrosstalkSources_1101468_52.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/99095763-3998-49bb-bdc9-60307b3414ae_isrForCrosstalkSources_1101468_52.stderr parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81421277ca076765e7bccfb1ba385519 with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/43aa8481-1b1a-4c39-81c1-b6f6c43394a2_isrForCrosstalkSources_1101468_47.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/43aa8481-1b1a-4c39-81c1-b6f6c43394a2_isrForCrosstalkSources_1101468_47.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dda1d056b3357d3bcc96dd6c62b96f90 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/4057f1b4-5e86-441f-810c-5c72fa5d15c8_isrForCrosstalkSources_1101471_16.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/4057f1b4-5e86-441f-810c-5c72fa5d15c8_isrForCrosstalkSources_1101471_16.stderr 2024-03-19 00:40:14 proc_lsst.multi:170 [INFO] [multi] job_id 17226010 proc_lsst.multi INFO: [multi] job_id 17226010 2024-03-19 00:40:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-19 00:40:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-19 00:40:14 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17226010 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17226010 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 17226010 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23453085687824 2024-03-19 00:40:14 proc_lsst.multi:146 [INFO] found job 96275 in provider local proc_lsst.multi INFO: found job 96275 in provider local 2024-03-19 00:40:14 proc_lsst.multi:146 [INFO] found job 17225950 in provider astro proc_lsst.multi INFO: found job 17225950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17b1f798e32ec32b1238ac26854097a7 with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/f76dc2e8-d41d-4059-8685-706603dc7f6f_isrForCrosstalkSources_1101465_6.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/f76dc2e8-d41d-4059-8685-706603dc7f6f_isrForCrosstalkSources_1101465_6.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6995bddfac0b6b48247f6a1527107339 with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/f9376738-a2b3-4ddb-8a63-01aaa94c6b8f_isrForCrosstalkSources_1101471_54.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/f9376738-a2b3-4ddb-8a63-01aaa94c6b8f_isrForCrosstalkSources_1101471_54.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 070aa13d051aff44a9f6ee7cc75485e3 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/9a114a1e-af9c-4ce7-8157-547db3117173_isrForCrosstalkSources_1101468_16.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/9a114a1e-af9c-4ce7-8157-547db3117173_isrForCrosstalkSources_1101468_16.stderr parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 684e93369139559b91854b206e1233f7 with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/0546b155-36d4-49c5-9aa3-df3021562e8e_isrForCrosstalkSources_1101471_62.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/0546b155-36d4-49c5-9aa3-df3021562e8e_isrForCrosstalkSources_1101471_62.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37b22db0d03493219939e00d52d8e625 with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/549bb2a6-6d3e-4ad9-8db3-b58eeec93404_isrForCrosstalkSources_1101465_56.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/549bb2a6-6d3e-4ad9-8db3-b58eeec93404_isrForCrosstalkSources_1101465_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:16 proc_lsst.multi:146 [INFO] found job 17225951 in provider astro proc_lsst.multi INFO: found job 17225951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cabaf4469b2c8cf3c18c821c3e16186 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/9740d960-b94f-4b67-bf91-683995b39183_isrForCrosstalkSources_1101471_22.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/9740d960-b94f-4b67-bf91-683995b39183_isrForCrosstalkSources_1101471_22.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01d34d61779f089433285c452a15e01a with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/5890ae21-0327-4b2c-b5e9-9681764cd56d_isrForCrosstalkSources_1101465_45.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/5890ae21-0327-4b2c-b5e9-9681764cd56d_isrForCrosstalkSources_1101465_45.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42d007bfc0bf543c8aa55c1d6b8c6467 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/482908ac-6494-4a64-b33e-495187008ba7_isrForCrosstalkSources_1101465_26.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/482908ac-6494-4a64-b33e-495187008ba7_isrForCrosstalkSources_1101465_26.stderr parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8991ce4eec96388353cd7470b1c7e1b4 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/36fe95ec-ba29-46a3-a801-e331eec48f87_isrForCrosstalkSources_1101465_53.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/36fe95ec-ba29-46a3-a801-e331eec48f87_isrForCrosstalkSources_1101465_53.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a22f35c4a107a1f5297f49dc7a394ca with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/8d41c1df-f35e-4606-9517-3fef04cf1e74_isrForCrosstalkSources_1101471_61.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/8d41c1df-f35e-4606-9517-3fef04cf1e74_isrForCrosstalkSources_1101471_61.stderr parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b62f94aacd03b6807d12d712d5692c9 with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/782434b0-daac-45ae-87a7-298c2a6bcf3d_isrForCrosstalkSources_1101465_52.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/782434b0-daac-45ae-87a7-298c2a6bcf3d_isrForCrosstalkSources_1101465_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:17 proc_lsst.multi:146 [INFO] found job 17225952 in provider astro proc_lsst.multi INFO: found job 17225952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 510f18651658e9f22e4b6255b5ed1ef5 with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/903a7008-c2a8-4daa-9c10-df2a909b036f_isrForCrosstalkSources_1101471_27.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/903a7008-c2a8-4daa-9c10-df2a909b036f_isrForCrosstalkSources_1101471_27.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffd4cdd2edb74553065ddb38f5c33707 with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/69053a71-4b5c-46b0-aa09-fabbe3908137_isrForCrosstalkSources_1101474_34.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/69053a71-4b5c-46b0-aa09-fabbe3908137_isrForCrosstalkSources_1101474_34.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88ece6163fb105cbc5f83b22837e340b with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/523a2f26-f1b6-4938-b25e-b60433150f09_isrForCrosstalkSources_1101468_44.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/523a2f26-f1b6-4938-b25e-b60433150f09_isrForCrosstalkSources_1101468_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:18 proc_lsst.multi:146 [INFO] found job 17225953 in provider astro proc_lsst.multi INFO: found job 17225953 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1704c0968c56218f12e114307153ae98 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/89f0fe78-60c7-446e-9d64-1e07e8962d43_isrForCrosstalkSources_1101471_8.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/89f0fe78-60c7-446e-9d64-1e07e8962d43_isrForCrosstalkSources_1101471_8.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a708313ff971fb2734aaa6c5b1f6b85b with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/647ed4cd-7f35-4e3b-8ef3-51ef3c2e62f3_isrForCrosstalkSources_1101474_8.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/647ed4cd-7f35-4e3b-8ef3-51ef3c2e62f3_isrForCrosstalkSources_1101474_8.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dee27fb0f008066be319faafb9b998ab with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/01cc464d-1c9c-4f97-a512-abb8c006021d_isrForCrosstalkSources_1101471_57.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/01cc464d-1c9c-4f97-a512-abb8c006021d_isrForCrosstalkSources_1101471_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:19 proc_lsst.multi:146 [INFO] found job 17225954 in provider astro proc_lsst.multi INFO: found job 17225954 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c403c73574ab0cce5b56a45ecb830aa5 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/259cc124-e11e-4dae-8011-a6c9bc7cab64_isrForCrosstalkSources_1101468_36.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/259cc124-e11e-4dae-8011-a6c9bc7cab64_isrForCrosstalkSources_1101468_36.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f46225fb994bbceb0c5e3e1bcbce1814 with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/fee0277c-ef15-4385-8180-9b8b98d5147f_isrForCrosstalkSources_1101471_6.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/fee0277c-ef15-4385-8180-9b8b98d5147f_isrForCrosstalkSources_1101471_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:21 proc_lsst.multi:146 [INFO] found job 17225955 in provider astro proc_lsst.multi INFO: found job 17225955 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1a595b0ff3eeff0f58eeee0891209e7 with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/1ab10ea8-8f62-4b10-a9ba-10ec761d4464_isrForCrosstalkSources_1101471_53.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/1ab10ea8-8f62-4b10-a9ba-10ec761d4464_isrForCrosstalkSources_1101471_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:22 proc_lsst.multi:146 [INFO] found job 17225956 in provider astro proc_lsst.multi INFO: found job 17225956 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ebcd97ff1c0f3576a7a914675ab7cbc with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/0586009a-1350-4dfd-a341-7e37197cd4fc_isrForCrosstalkSources_1101474_30.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/0586009a-1350-4dfd-a341-7e37197cd4fc_isrForCrosstalkSources_1101474_30.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1470d7e1e8b63e67f74a5f4e2102679 with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/75271aa6-adf2-4965-a288-82e0247a15e4_isrForCrosstalkSources_1101474_19.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/75271aa6-adf2-4965-a288-82e0247a15e4_isrForCrosstalkSources_1101474_19.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8a3042b6f5cf6f83ca2e74d8e76dbee with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/f7c34b97-759f-486e-b768-7fa7cae9e33c_isrForCrosstalkSources_1101474_46.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/f7c34b97-759f-486e-b768-7fa7cae9e33c_isrForCrosstalkSources_1101474_46.stderr parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44e4ddddc2c0fc4876291cdf4a50267d with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/c2b6064c-5abe-4cd9-b3ce-1d766e6f61fd_isrForCrosstalkSources_1101468_2.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/c2b6064c-5abe-4cd9-b3ce-1d766e6f61fd_isrForCrosstalkSources_1101468_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:23 proc_lsst.multi:146 [INFO] found job 17225957 in provider astro proc_lsst.multi INFO: found job 17225957 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 175050acdf839408929bb7dc199e86df with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/4abd2c0c-e7ca-4f94-bf19-fa5645a42838_isrForCrosstalkSources_1101474_44.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/4abd2c0c-e7ca-4f94-bf19-fa5645a42838_isrForCrosstalkSources_1101474_44.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcc4ef207ba4a4db44985d536bb6fa10 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/e8b4dd7b-6a8f-460d-acbc-1cbd4fe8ee51_isrForCrosstalkSources_1101474_25.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/e8b4dd7b-6a8f-460d-acbc-1cbd4fe8ee51_isrForCrosstalkSources_1101474_25.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8b80cf7420289881cbf244cf95d6a73 with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/1334c6b8-9498-4e38-aa76-a37b2a8a4127_isrForCrosstalkSources_1101468_38.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/1334c6b8-9498-4e38-aa76-a37b2a8a4127_isrForCrosstalkSources_1101468_38.stderr parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbf2829d8f99ff97c9fd4d3c9fbcaed9 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/6e43fe3d-c04c-4757-91c2-48f9e4c60a23_isrForCrosstalkSources_1101471_24.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/6e43fe3d-c04c-4757-91c2-48f9e4c60a23_isrForCrosstalkSources_1101471_24.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b41f3e923e817743c0ab63212b29fde9 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/5179c732-2584-4925-80ca-7e71aa85d1ca_isrForCrosstalkSources_1101474_31.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/5179c732-2584-4925-80ca-7e71aa85d1ca_isrForCrosstalkSources_1101474_31.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0eee7e23561e27223ccb6210de7d4c1 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/492d9121-fa1d-4c0e-a542-990a391c8e6e_isrForCrosstalkSources_1101471_30.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/492d9121-fa1d-4c0e-a542-990a391c8e6e_isrForCrosstalkSources_1101471_30.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d6691598304c5d3b93058ac66b02e85 with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/81ac2801-b35c-48d7-97ae-4077cc391f22_isrForCrosstalkSources_1101465_24.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/81ac2801-b35c-48d7-97ae-4077cc391f22_isrForCrosstalkSources_1101465_24.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 812c18e9785b68511fd6cd4214080d76 with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/3679b6db-77d9-48e9-85cf-510fb205cfbb_isrForCrosstalkSources_1101465_34.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/3679b6db-77d9-48e9-85cf-510fb205cfbb_isrForCrosstalkSources_1101465_34.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d1a2553f9a217d7d83e4ed25cd04856 with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/582d8fd0-16cd-46df-88ac-29ad2430e9e7_isrForCrosstalkSources_1101471_52.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/582d8fd0-16cd-46df-88ac-29ad2430e9e7_isrForCrosstalkSources_1101471_52.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry def362ded478807539e8734fe6d8ba18 with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/4b69004d-d99e-4634-a04c-843ab62d2141_isrForCrosstalkSources_1101468_30.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/4b69004d-d99e-4634-a04c-843ab62d2141_isrForCrosstalkSources_1101468_30.stderr parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 032bc453c8cf0f0d2c8d16bbb15e96f5 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/20fe314a-6113-4037-a3d6-d7d1ee2cc33e_isrForCrosstalkSources_1101468_26.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/20fe314a-6113-4037-a3d6-d7d1ee2cc33e_isrForCrosstalkSources_1101468_26.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3d902ca90fb36294d213f636effd7bf with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/5d091e6f-4ebd-41ce-8e9f-af7eeab0f090_isrForCrosstalkSources_1101465_8.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/5d091e6f-4ebd-41ce-8e9f-af7eeab0f090_isrForCrosstalkSources_1101465_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:24 proc_lsst.multi:146 [INFO] found job 17225958 in provider astro proc_lsst.multi INFO: found job 17225958 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9b5ae2dfeda627f61ee1830b5e3ffc1 with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/c512f342-da05-4acf-8557-05c005e2b08f_isrForCrosstalkSources_1101471_21.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/c512f342-da05-4acf-8557-05c005e2b08f_isrForCrosstalkSources_1101471_21.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8882f7864561cf515b308a678f0845f0 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/42c206e0-1229-485d-b964-2a233b519601_isrForCrosstalkSources_1101474_12.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/42c206e0-1229-485d-b964-2a233b519601_isrForCrosstalkSources_1101474_12.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50285f6496a77866f1d80323de5da755 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/701d4dd4-421a-4c73-b636-54ffac82f949_isrForCrosstalkSources_1101474_13.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/701d4dd4-421a-4c73-b636-54ffac82f949_isrForCrosstalkSources_1101474_13.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58de3aea9b3f242325bdbea87a14d97f with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/a4105d02-01bc-4e75-abab-e2c599a7d52b_isrForCrosstalkSources_1101474_54.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/a4105d02-01bc-4e75-abab-e2c599a7d52b_isrForCrosstalkSources_1101474_54.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9845013f5f3da5789f5bfd52f4e8c314 with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/ab8ade4f-3644-4a02-832e-0e9ad1597cef_isrForCrosstalkSources_1101471_20.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/ab8ade4f-3644-4a02-832e-0e9ad1597cef_isrForCrosstalkSources_1101471_20.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 040609dc965fef06eaf7cfd9136f0fe4 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/f44b483d-6bdf-4d1d-a864-e396f67d8201_isrForCrosstalkSources_1101465_20.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/f44b483d-6bdf-4d1d-a864-e396f67d8201_isrForCrosstalkSources_1101465_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:25 proc_lsst.multi:146 [INFO] found job 17225959 in provider astro proc_lsst.multi INFO: found job 17225959 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c443f88208ac4072ace445f071a9f3db with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/55f8f94d-10fe-432e-8b0c-1e4c2100719f_isrForCrosstalkSources_1101471_59.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/55f8f94d-10fe-432e-8b0c-1e4c2100719f_isrForCrosstalkSources_1101471_59.stderr parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c4accf1841c1b567856b712d457c7a6 with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/5ce09d38-3a05-4424-97db-e310e7563dc6_isrForCrosstalkSources_1101465_48.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/5ce09d38-3a05-4424-97db-e310e7563dc6_isrForCrosstalkSources_1101465_48.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a769650cc60cf0668827ea59ee72d5dc with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/47deb65e-c865-4814-a00e-9cfc903e025b_isrForCrosstalkSources_1101465_10.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/47deb65e-c865-4814-a00e-9cfc903e025b_isrForCrosstalkSources_1101465_10.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67306bcc5da85ca154ab7b6006a553f7 with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/8ae13032-c98a-4d27-a126-b9c1f3b66da6_isrForCrosstalkSources_1101465_37.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/8ae13032-c98a-4d27-a126-b9c1f3b66da6_isrForCrosstalkSources_1101465_37.stderr parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a51476e464e6ac1c18353ecd8094b4ab with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/ec5a4bdd-ce36-4184-87f1-d0432eac7ad7_isrForCrosstalkSources_1101468_46.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/ec5a4bdd-ce36-4184-87f1-d0432eac7ad7_isrForCrosstalkSources_1101468_46.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec582020d17bc06ca817a6dc20ccd0fb with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/4266bc6d-d2d6-4481-8143-be12b70b850b_isrForCrosstalkSources_1101471_11.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/4266bc6d-d2d6-4481-8143-be12b70b850b_isrForCrosstalkSources_1101471_11.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdff33c37a1c830984bb23c28eb2d43a with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/22bd586f-eaf1-4aae-bd47-b3749db0bf91_isrForCrosstalkSources_1101471_36.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/22bd586f-eaf1-4aae-bd47-b3749db0bf91_isrForCrosstalkSources_1101471_36.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee550d709272eb86259349c154210dfd with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/a4860be9-da49-4e97-a086-0fcbac27ae4c_isrForCrosstalkSources_1101471_37.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/a4860be9-da49-4e97-a086-0fcbac27ae4c_isrForCrosstalkSources_1101471_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:26 proc_lsst.multi:146 [INFO] found job 17225960 in provider astro proc_lsst.multi INFO: found job 17225960 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bd6c95d4d5f6be2da3cdcef05f36edb with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/fdc2501e-e61c-46ee-9553-3d5069604d75_isrForCrosstalkSources_1101465_59.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/fdc2501e-e61c-46ee-9553-3d5069604d75_isrForCrosstalkSources_1101465_59.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72bc551c69695acc5fc1698c820cba15 with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/accfbb72-aaa2-4a81-b43c-e63ada5a9cbf_isrForCrosstalkSources_1101465_61.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/accfbb72-aaa2-4a81-b43c-e63ada5a9cbf_isrForCrosstalkSources_1101465_61.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4f130ae4c17861f9b374868171a50f3 with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/5dc5bfcf-f106-4b8a-9038-3d353714bec7_isrForCrosstalkSources_1101471_47.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/5dc5bfcf-f106-4b8a-9038-3d353714bec7_isrForCrosstalkSources_1101471_47.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45018b781a8a4e19443619ad3a611f90 with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/cbedec63-accf-4811-9c01-d1f8ac868672_isrForCrosstalkSources_1101465_54.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/cbedec63-accf-4811-9c01-d1f8ac868672_isrForCrosstalkSources_1101465_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:26 proc_lsst.multi:146 [INFO] found job 17225961 in provider astro proc_lsst.multi INFO: found job 17225961 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e665981d72604b11e603d4e8d68c060f with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/ebef071b-4c2b-4317-bb3d-e8c17debd107_isrForCrosstalkSources_1101474_5.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/ebef071b-4c2b-4317-bb3d-e8c17debd107_isrForCrosstalkSources_1101474_5.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d82dd32d2fdd53e4503e931f19dda8a1 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/38927832-d5fa-477e-b0b2-ed5e4f2f9907_isrForCrosstalkSources_1101465_55.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/38927832-d5fa-477e-b0b2-ed5e4f2f9907_isrForCrosstalkSources_1101465_55.stderr parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d140cb744277f68d604ddab150bdaeae with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/a7bcd230-bf57-453c-9e39-7c12dd37a53e_isrForCrosstalkSources_1101474_3.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/a7bcd230-bf57-453c-9e39-7c12dd37a53e_isrForCrosstalkSources_1101474_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:27 proc_lsst.multi:146 [INFO] found job 17225962 in provider astro proc_lsst.multi INFO: found job 17225962 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70ad9c6d5da3f8e3f57e1ff00fc94a77 with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/683411b6-dae4-45ce-be15-6a428032cab1_isrForCrosstalkSources_1101474_42.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/683411b6-dae4-45ce-be15-6a428032cab1_isrForCrosstalkSources_1101474_42.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e56dc08065f2ba98b34b8e579a0db148 with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/61eab9f2-22e1-492a-9ca9-e8c1f474f1d6_isrForCrosstalkSources_1101474_53.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/61eab9f2-22e1-492a-9ca9-e8c1f474f1d6_isrForCrosstalkSources_1101474_53.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56a3ac61a747de9a159971a96969b8fc with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/0c99a475-37d5-4c78-8b85-340f61f13789_isrForCrosstalkSources_1101471_40.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/0c99a475-37d5-4c78-8b85-340f61f13789_isrForCrosstalkSources_1101471_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:27 proc_lsst.multi:146 [INFO] found job 17225963 in provider astro proc_lsst.multi INFO: found job 17225963 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:27 proc_lsst.multi:146 [INFO] found job 17225964 in provider astro proc_lsst.multi INFO: found job 17225964 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:27 proc_lsst.multi:146 [INFO] found job 17225965 in provider astro proc_lsst.multi INFO: found job 17225965 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:28 proc_lsst.multi:146 [INFO] found job 17225966 in provider astro proc_lsst.multi INFO: found job 17225966 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:28 proc_lsst.multi:146 [INFO] found job 17225967 in provider astro proc_lsst.multi INFO: found job 17225967 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:29 proc_lsst.multi:146 [INFO] found job 17225968 in provider astro proc_lsst.multi INFO: found job 17225968 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:29 proc_lsst.multi:146 [INFO] found job 17225969 in provider astro proc_lsst.multi INFO: found job 17225969 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 885e5f176ed1086d591e34c28d6d84b7 with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/69070fa0-10cd-4a5f-93e1-73997fbcc03d_isrForCrosstalkSources_1101471_1.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/69070fa0-10cd-4a5f-93e1-73997fbcc03d_isrForCrosstalkSources_1101471_1.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00cfdfe816eef1f685b5811e2db49471 with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/a89be5eb-1228-4cdc-9553-c22af674fc43_isrForCrosstalkSources_1101471_50.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/a89be5eb-1228-4cdc-9553-c22af674fc43_isrForCrosstalkSources_1101471_50.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f5fba32e1cd10faa88bb60230fa7caa with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/1cbbd28b-1dc5-4d8f-8322-0c6ba858349d_isrForCrosstalkSources_1101468_43.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/1cbbd28b-1dc5-4d8f-8322-0c6ba858349d_isrForCrosstalkSources_1101468_43.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b97eecfbfce384e722fce6e09f3a08d1 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/f922b3e9-1763-4e82-8c4b-5db1f00ef4b3_isrForCrosstalkSources_1101468_42.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/f922b3e9-1763-4e82-8c4b-5db1f00ef4b3_isrForCrosstalkSources_1101468_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:30 proc_lsst.multi:146 [INFO] found job 17225970 in provider astro proc_lsst.multi INFO: found job 17225970 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:30 proc_lsst.multi:146 [INFO] found job 17225971 in provider astro proc_lsst.multi INFO: found job 17225971 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a27d9208d0b97961e489a97a971fd568 with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/06a2f99d-9c0a-46d9-ab09-7023dcdd6c86_isrForCrosstalkSources_1101474_23.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/06a2f99d-9c0a-46d9-ab09-7023dcdd6c86_isrForCrosstalkSources_1101474_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:30 proc_lsst.multi:146 [INFO] found job 17225972 in provider astro proc_lsst.multi INFO: found job 17225972 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:31 proc_lsst.multi:146 [INFO] found job 17225973 in provider astro proc_lsst.multi INFO: found job 17225973 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:31 proc_lsst.multi:146 [INFO] found job 17225974 in provider astro proc_lsst.multi INFO: found job 17225974 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:31 proc_lsst.multi:146 [INFO] found job 17225975 in provider astro proc_lsst.multi INFO: found job 17225975 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:31 proc_lsst.multi:146 [INFO] found job 17225976 in provider astro proc_lsst.multi INFO: found job 17225976 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3dcfb1af56c3064b27d71b58d203600d with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/a984c62e-ebad-4c15-b6c9-bc2be72e5079_isrForCrosstalkSources_1101471_56.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/a984c62e-ebad-4c15-b6c9-bc2be72e5079_isrForCrosstalkSources_1101471_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:31 proc_lsst.multi:146 [INFO] found job 17225977 in provider astro proc_lsst.multi INFO: found job 17225977 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:31 proc_lsst.multi:146 [INFO] found job 17225978 in provider astro proc_lsst.multi INFO: found job 17225978 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:32 proc_lsst.multi:146 [INFO] found job 17225979 in provider astro proc_lsst.multi INFO: found job 17225979 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:32 proc_lsst.multi:146 [INFO] found job 17225980 in provider escience_rtx6k proc_lsst.multi INFO: found job 17225980 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225980,17225981,17225982,17225983,17225984' parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8abf748f8fdaac59b956ae45cde5345d with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/edc9a63a-430d-4b7b-8e95-7615e2c9534b_isrForCrosstalkSources_1101465_23.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/edc9a63a-430d-4b7b-8e95-7615e2c9534b_isrForCrosstalkSources_1101465_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225983 R 17225984 R 17225981 R 17225982 R 17225980 R parsl.providers.slurm.slurm DEBUG: Updating job 17225983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225980 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:32 proc_lsst.multi:146 [INFO] found job 17225981 in provider escience_rtx6k proc_lsst.multi INFO: found job 17225981 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225980,17225981,17225982,17225983,17225984' parsl.providers.slurm.slurm DEBUG: squeue returned 17225983 R 17225984 R 17225981 R 17225982 R 17225980 R parsl.providers.slurm.slurm DEBUG: Updating job 17225983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225980 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:32 proc_lsst.multi:146 [INFO] found job 17225982 in provider escience_rtx6k proc_lsst.multi INFO: found job 17225982 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225980,17225981,17225982,17225983,17225984' parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa9d62d355e01b7b228c8de24464cff2 with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/d8efc727-5cb4-45a1-b86c-3ab34826ee05_isrForCrosstalkSources_1101471_35.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/d8efc727-5cb4-45a1-b86c-3ab34826ee05_isrForCrosstalkSources_1101471_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225983 R 17225984 R 17225981 R 17225982 R 17225980 R parsl.providers.slurm.slurm DEBUG: Updating job 17225983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225980 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:33 proc_lsst.multi:146 [INFO] found job 17225983 in provider escience_rtx6k proc_lsst.multi INFO: found job 17225983 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225980,17225981,17225982,17225983,17225984' parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f71a013c89fbee6973dc0224bf03299 with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/752dd2f1-6a80-4bed-88e3-70f274f3cbdd_isrForCrosstalkSources_1101471_55.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/752dd2f1-6a80-4bed-88e3-70f274f3cbdd_isrForCrosstalkSources_1101471_55.stderr parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46ae3ef52c2e2379e12a253c24f2d399 with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/71a15409-e386-4453-91d7-a5d4dac5529c_isrForCrosstalkSources_1101474_41.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/71a15409-e386-4453-91d7-a5d4dac5529c_isrForCrosstalkSources_1101474_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225983 R 17225984 R 17225981 R 17225982 R 17225980 R parsl.providers.slurm.slurm DEBUG: Updating job 17225983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225980 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:34 proc_lsst.multi:146 [INFO] found job 17225984 in provider escience_rtx6k proc_lsst.multi INFO: found job 17225984 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225980,17225981,17225982,17225983,17225984' parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18c175d01d33fb3f5dbe989db7b2e260 with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/d3a15c41-5791-4b06-9976-e0d1d22c80f0_isrForCrosstalkSources_1101468_31.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/d3a15c41-5791-4b06-9976-e0d1d22c80f0_isrForCrosstalkSources_1101468_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225983 R 17225984 R 17225981 R 17225982 R 17225980 R parsl.providers.slurm.slurm DEBUG: Updating job 17225983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225980 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:36 proc_lsst.multi:146 [INFO] found job 17225985 in provider escience_a40 proc_lsst.multi INFO: found job 17225985 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64ac6393f98c1b09dc647c2aa8487860 with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/2ec4d6d1-3edb-4d0a-9ef0-59cf3163b308_isrForCrosstalkSources_1101468_19.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/2ec4d6d1-3edb-4d0a-9ef0-59cf3163b308_isrForCrosstalkSources_1101468_19.stderr parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c27a66b2535a8db54be78270913c46b with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/b705e6c1-ecac-4e0a-b1f3-580b7e4400e4_isrForCrosstalkSources_1101468_9.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/b705e6c1-ecac-4e0a-b1f3-580b7e4400e4_isrForCrosstalkSources_1101468_9.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b26a8491f46065543bef940faeb7805 with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/31bfff51-c18d-4623-bb51-270de67adc56_isrForCrosstalkSources_1101468_1.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/31bfff51-c18d-4623-bb51-270de67adc56_isrForCrosstalkSources_1101468_1.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99febddd9526120dd0997251e87bfbae with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/4a4efa3f-e8ea-492e-9dcb-9d6d4c81d5eb_isrForCrosstalkSources_1101471_34.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/4a4efa3f-e8ea-492e-9dcb-9d6d4c81d5eb_isrForCrosstalkSources_1101471_34.stderr parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 101790b7dc2eb1a688f0018cec865508 with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/962ba212-e1df-4477-89c8-a7829d017fbb_isrForCrosstalkSources_1101471_33.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/962ba212-e1df-4477-89c8-a7829d017fbb_isrForCrosstalkSources_1101471_33.stderr parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67854befaa9700d05c3c5f6a0b9fe1d9 with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/0c5868dc-f790-4be4-95bd-bbcd17ce6a60_isrForCrosstalkSources_1101468_18.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/0c5868dc-f790-4be4-95bd-bbcd17ce6a60_isrForCrosstalkSources_1101468_18.stderr parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5464f3a086d936be6970370adca4a9eb with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/73f59019-17ec-46c9-ad97-273bfdc4372c_isrForCrosstalkSources_1101468_50.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/73f59019-17ec-46c9-ad97-273bfdc4372c_isrForCrosstalkSources_1101468_50.stderr parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c5fae4914c05eb7af5a96fb0d3c7ec3 with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/6bced9ee-24f8-4d23-870e-6582c73f426b_isrForCrosstalkSources_1101468_6.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/6bced9ee-24f8-4d23-870e-6582c73f426b_isrForCrosstalkSources_1101468_6.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92a7991bfd4cb926c423a1ca7438b12e with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/dceac4e5-bf3f-4ade-ab07-1f11b1dc5a4f_isrForCrosstalkSources_1101468_24.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/dceac4e5-bf3f-4ade-ab07-1f11b1dc5a4f_isrForCrosstalkSources_1101468_24.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55663dea15b224e94640d7c2c9cf8bac with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/48aad19f-1a13-41b0-8a14-7ce83dc89be0_isrForCrosstalkSources_1101471_13.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/48aad19f-1a13-41b0-8a14-7ce83dc89be0_isrForCrosstalkSources_1101471_13.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2898d9769eeacea01b158aec5bcab89 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/b66d41a4-d165-4b2d-a8fb-85e28b5435d7_isrForCrosstalkSources_1101465_13.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/b66d41a4-d165-4b2d-a8fb-85e28b5435d7_isrForCrosstalkSources_1101465_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:38 proc_lsst.multi:146 [INFO] found job 17225986 in provider escience_a40 proc_lsst.multi INFO: found job 17225986 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3134d7d8d65b3d3c60a768ea445cf8c with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/a226bb6c-cd76-4614-adc7-45ca5c6d4091_isrForCrosstalkSources_1101465_35.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/a226bb6c-cd76-4614-adc7-45ca5c6d4091_isrForCrosstalkSources_1101465_35.stderr parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5c14e3f19807a0c2cee3996d6ec5eb5 with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/6e65604b-3861-4047-9d21-6e6ae5da7f34_isrForCrosstalkSources_1101468_56.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/6e65604b-3861-4047-9d21-6e6ae5da7f34_isrForCrosstalkSources_1101468_56.stderr parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0737b97a202e77ace83a1b0a6d2084ea with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/4043b8d8-0173-4bb8-be14-f2e787f773be_isrForCrosstalkSources_1101465_18.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/4043b8d8-0173-4bb8-be14-f2e787f773be_isrForCrosstalkSources_1101465_18.stderr parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e57b05315821017c4870c6cb904007c with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/1da1ee71-2cdc-436d-9fff-093d6fbce011_isrForCrosstalkSources_1101474_37.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/1da1ee71-2cdc-436d-9fff-093d6fbce011_isrForCrosstalkSources_1101474_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:39 proc_lsst.multi:146 [INFO] found job 17225987 in provider escience_a40 proc_lsst.multi INFO: found job 17225987 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 408f60b4ab53c3fa4749410df62a5a77 with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/c6f0416f-d221-405f-a81e-162657cd967d_isrForCrosstalkSources_1101474_14.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/c6f0416f-d221-405f-a81e-162657cd967d_isrForCrosstalkSources_1101474_14.stderr parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da205d5ffa25f3e752a060dea63522b5 with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/f1cf5f9d-11c9-499a-943c-52ea1738788c_isrForCrosstalkSources_1101474_57.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/f1cf5f9d-11c9-499a-943c-52ea1738788c_isrForCrosstalkSources_1101474_57.stderr parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71766837f67c2016f659a7dce15329e7 with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/5dc00973-54a8-4828-a3f1-7cdeed7f5648_isrForCrosstalkSources_1101474_29.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/5dc00973-54a8-4828-a3f1-7cdeed7f5648_isrForCrosstalkSources_1101474_29.stderr parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36fa9994ff5087eca35129ba06af67da with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/d43d3882-9d6e-421d-9913-7310d4a24a79_isrForCrosstalkSources_1101471_28.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/d43d3882-9d6e-421d-9913-7310d4a24a79_isrForCrosstalkSources_1101471_28.stderr parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 920711a5906a9f93a9e6dc51bdde051f with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/e2f9340a-2960-47a1-a420-0249bdd01261_isrForCrosstalkSources_1101468_5.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/e2f9340a-2960-47a1-a420-0249bdd01261_isrForCrosstalkSources_1101468_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:41 proc_lsst.multi:146 [INFO] found job 17225988 in provider escience_a40 proc_lsst.multi INFO: found job 17225988 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3d420e3c374c6101b6405383885788f with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/16491f04-ce38-4ab0-acfe-837b010b9805_isrForCrosstalkSources_1101468_25.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/16491f04-ce38-4ab0-acfe-837b010b9805_isrForCrosstalkSources_1101468_25.stderr parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 274c71b0fa117efa3f280a6e0d88f042 with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/5a5828aa-4632-4d95-8e40-b9a95176f6c8_isrForCrosstalkSources_1101471_60.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/5a5828aa-4632-4d95-8e40-b9a95176f6c8_isrForCrosstalkSources_1101471_60.stderr parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8236a1e26cad853926c9289deb78ba85 with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/31ee2544-d886-4255-bde5-2a80679a8ad5_isrForCrosstalkSources_1101468_33.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/31ee2544-d886-4255-bde5-2a80679a8ad5_isrForCrosstalkSources_1101468_33.stderr parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 205ecc6cb2ad675ad19dfa84ad820324 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/80eb1844-0b61-4589-b727-52575fa13afe_isrForCrosstalkSources_1101471_25.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/80eb1844-0b61-4589-b727-52575fa13afe_isrForCrosstalkSources_1101471_25.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec4b8ce5b29e8f34f7f69d9953a3de79 with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/9cd03a18-34c9-437c-a4e2-1a6f8324376d_isrForCrosstalkSources_1101471_32.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/9cd03a18-34c9-437c-a4e2-1a6f8324376d_isrForCrosstalkSources_1101471_32.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8661c692d7d161947366b5a4df60f0d2 with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/65b60993-b6fb-4795-8af9-e9d571ed0cbd_isrForCrosstalkSources_1101471_39.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/65b60993-b6fb-4795-8af9-e9d571ed0cbd_isrForCrosstalkSources_1101471_39.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d2bd9a8a5ff75e17ae68a7c92772ebe with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/7a1ea6be-280d-4fed-bafe-17e926d73bee_isrForCrosstalkSources_1101465_14.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/7a1ea6be-280d-4fed-bafe-17e926d73bee_isrForCrosstalkSources_1101465_14.stderr parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a5978082fa16c66e9c4af3d21829c14 with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/e7224858-1ee7-4866-87aa-56663677221d_isrForCrosstalkSources_1101468_13.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/e7224858-1ee7-4866-87aa-56663677221d_isrForCrosstalkSources_1101468_13.stderr parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e83e777cbb1ac51fa55063385fe7de2f with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/cae7b685-5066-40a7-8254-a2181d981804_isrForCrosstalkSources_1101465_40.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/cae7b685-5066-40a7-8254-a2181d981804_isrForCrosstalkSources_1101465_40.stderr parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5eaf21a634521aee96262b78d01172f2 with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/1fcffecb-6536-4f9e-a1bf-a6d4848736a8_isrForCrosstalkSources_1101465_27.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/1fcffecb-6536-4f9e-a1bf-a6d4848736a8_isrForCrosstalkSources_1101465_27.stderr parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22566847265b0bb45c60d3e9fb34b6d1 with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/a993ba8a-f991-4b24-83ee-89e82408361c_isrForCrosstalkSources_1101474_40.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/a993ba8a-f991-4b24-83ee-89e82408361c_isrForCrosstalkSources_1101474_40.stderr parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99d2dc9ec16895a9035c9e424c831ba2 with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/44090055-d331-41c9-a220-cb54648e2321_isrForCrosstalkSources_1101471_23.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/44090055-d331-41c9-a220-cb54648e2321_isrForCrosstalkSources_1101471_23.stderr parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64d8c654bacea4ce69d6f198f4750937 with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/31895b69-bd81-44db-9714-cbda07db3486_isrForCrosstalkSources_1101474_7.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/31895b69-bd81-44db-9714-cbda07db3486_isrForCrosstalkSources_1101474_7.stderr parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f782bcf8fcbedefac7bc823674027a5e with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/22a7d9b5-2185-4c34-90c5-9bb52ca6afbb_isrForCrosstalkSources_1101474_16.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/22a7d9b5-2185-4c34-90c5-9bb52ca6afbb_isrForCrosstalkSources_1101474_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:42 proc_lsst.multi:146 [INFO] found job 17225989 in provider escience_a40 proc_lsst.multi INFO: found job 17225989 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad75176afa5ceeda90790576bbcf4dad with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/b008ab9b-b459-4edf-a4b4-0d2a0d160257_isrForCrosstalkSources_1101474_10.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/b008ab9b-b459-4edf-a4b4-0d2a0d160257_isrForCrosstalkSources_1101474_10.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7323a75a85ba98ec1dfdfe8ae586bbb2 with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/91de3360-d6d5-4cdc-b738-30d576628673_isrForCrosstalkSources_1101468_54.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/91de3360-d6d5-4cdc-b738-30d576628673_isrForCrosstalkSources_1101468_54.stderr parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f127f5091847dee4fd77e507147e8ec with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/74ea063a-ab74-466e-96db-3b7028a5641b_isrForCrosstalkSources_1101468_7.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/74ea063a-ab74-466e-96db-3b7028a5641b_isrForCrosstalkSources_1101468_7.stderr parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 667b52e40062f5f5255a1786578502be with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/aa605f35-8d39-4bb3-a42e-155a50e9d3c2_isrForCrosstalkSources_1101471_29.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/aa605f35-8d39-4bb3-a42e-155a50e9d3c2_isrForCrosstalkSources_1101471_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:42 proc_lsst.multi:146 [INFO] found job 17225990 in provider escience_a40 proc_lsst.multi INFO: found job 17225990 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a17613dfd389a0f2d8ee3b137fdba64 with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/c1b7471e-58f6-4e96-be99-90e337f76297_isrForCrosstalkSources_1101468_10.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/c1b7471e-58f6-4e96-be99-90e337f76297_isrForCrosstalkSources_1101468_10.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db0acdc6edabb10f41c9c0181c8ce600 with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/a1547728-e8b8-4aad-bd5e-722430704fe3_isrForCrosstalkSources_1101474_47.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/a1547728-e8b8-4aad-bd5e-722430704fe3_isrForCrosstalkSources_1101474_47.stderr parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83cc4d7337b4f35d0fa06179516f8c99 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/96d9ecd5-8430-4840-bf7b-b4d5d69d7352_isrForCrosstalkSources_1101468_41.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/96d9ecd5-8430-4840-bf7b-b4d5d69d7352_isrForCrosstalkSources_1101468_41.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be085e6cc3c9d56b502fe8db55412012 with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/5fad046e-3002-419a-91f4-22ed064c427f_isrForCrosstalkSources_1101474_59.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/5fad046e-3002-419a-91f4-22ed064c427f_isrForCrosstalkSources_1101474_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:43 proc_lsst.multi:146 [INFO] found job 17225991 in provider escience_a40 proc_lsst.multi INFO: found job 17225991 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44516ad9a75680084404675704b892f8 with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/1c313475-e32f-4f3c-b257-0a4c1baaab95_isrForCrosstalkSources_1101465_62.stdout parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/1c313475-e32f-4f3c-b257-0a4c1baaab95_isrForCrosstalkSources_1101465_62.stderr parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Storing app cache entry 6f653252901bb7f794bdb368743beeb0 with result from task 170 parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/cb0b5f46-9029-498e-bbfb-ec69861340c8_isrForCrosstalkSources_1101474_60.stdout parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/cb0b5f46-9029-498e-bbfb-ec69861340c8_isrForCrosstalkSources_1101474_60.stderr parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:43 proc_lsst.multi:146 [INFO] found job 17225992 in provider escience_a40 proc_lsst.multi INFO: found job 17225992 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:43 proc_lsst.multi:146 [INFO] found job 17225993 in provider escience_a40 proc_lsst.multi INFO: found job 17225993 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bac573818a60d7fa2aca68755c62fb21 with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/6916ace6-1003-4d30-996e-074f2b79a201_isrForCrosstalkSources_1101468_60.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/6916ace6-1003-4d30-996e-074f2b79a201_isrForCrosstalkSources_1101468_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:44 proc_lsst.multi:146 [INFO] found job 17225994 in provider escience_a40 proc_lsst.multi INFO: found job 17225994 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:44 proc_lsst.multi:146 [INFO] found job 17225995 in provider escience_a40 proc_lsst.multi INFO: found job 17225995 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:44 proc_lsst.multi:146 [INFO] found job 17225996 in provider escience_a40 proc_lsst.multi INFO: found job 17225996 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:44 proc_lsst.multi:146 [INFO] found job 17225997 in provider escience_a40 proc_lsst.multi INFO: found job 17225997 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbca22301eb17297b9c69645b72fe4d6 with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/a7eb4196-3a5d-4268-8296-4182c1bd34f9_isrForCrosstalkSources_1101465_29.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/a7eb4196-3a5d-4268-8296-4182c1bd34f9_isrForCrosstalkSources_1101465_29.stderr parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a20821b5cc75a5bd4efc73c3d088b1a with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/fae3e0f8-3ded-49d4-987e-7b8cdcbec387_isrForCrosstalkSources_1101465_16.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/fae3e0f8-3ded-49d4-987e-7b8cdcbec387_isrForCrosstalkSources_1101465_16.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 341b5a088027e676543e82a796c85832 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/d2c7da22-3f60-4d1e-a727-607eff679943_isrForCrosstalkSources_1101471_44.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/d2c7da22-3f60-4d1e-a727-607eff679943_isrForCrosstalkSources_1101471_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:44 proc_lsst.multi:146 [INFO] found job 17225998 in provider escience_a40 proc_lsst.multi INFO: found job 17225998 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33bf038f8293b863819996f963e46f70 with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/d8b6a722-d25d-48fd-973a-e8930822f98f_isrForCrosstalkSources_1101468_61.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/d8b6a722-d25d-48fd-973a-e8930822f98f_isrForCrosstalkSources_1101468_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:45 proc_lsst.multi:146 [INFO] found job 17225999 in provider escience_a40 proc_lsst.multi INFO: found job 17225999 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7667c18dbced6066f085b014d28646e2 with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/8c9c5b41-94f0-4453-a9a7-341b193734dd_isrForCrosstalkSources_1101465_44.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/8c9c5b41-94f0-4453-a9a7-341b193734dd_isrForCrosstalkSources_1101465_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:45 proc_lsst.multi:146 [INFO] found job 17226000 in provider escience_a40 proc_lsst.multi INFO: found job 17226000 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:46 proc_lsst.multi:146 [INFO] found job 17226001 in provider escience_a40 proc_lsst.multi INFO: found job 17226001 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d85b303cb8829b163449d42123839578 with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/4946a7e2-a828-4c0c-8c26-fd225f110141_isrForCrosstalkSources_1101465_17.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/4946a7e2-a828-4c0c-8c26-fd225f110141_isrForCrosstalkSources_1101465_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:46 proc_lsst.multi:146 [INFO] found job 17226002 in provider escience_a40 proc_lsst.multi INFO: found job 17226002 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:46 proc_lsst.multi:146 [INFO] found job 17226003 in provider escience_a40 proc_lsst.multi INFO: found job 17226003 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:46 proc_lsst.multi:146 [INFO] found job 17226004 in provider escience_a40 proc_lsst.multi INFO: found job 17226004 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:46 proc_lsst.multi:146 [INFO] found job 17226005 in provider checkpoint proc_lsst.multi INFO: found job 17226005 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226005,17226006,17226007,17226008,17226009,17226010' parsl.providers.slurm.slurm DEBUG: squeue returned 17226009 PD 17226010 PD 17226005 R 17226006 R 17226007 R 17226008 R parsl.providers.slurm.slurm DEBUG: Updating job 17226009 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226010 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226008 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:46 proc_lsst.multi:146 [INFO] found job 17226006 in provider checkpoint proc_lsst.multi INFO: found job 17226006 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226005,17226006,17226007,17226008,17226009,17226010' parsl.providers.slurm.slurm DEBUG: squeue returned 17226009 PD 17226010 PD 17226005 R 17226006 R 17226007 R 17226008 R parsl.providers.slurm.slurm DEBUG: Updating job 17226009 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226010 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226008 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:47 proc_lsst.multi:146 [INFO] found job 17226007 in provider checkpoint proc_lsst.multi INFO: found job 17226007 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226005,17226006,17226007,17226008,17226009,17226010' parsl.providers.slurm.slurm DEBUG: squeue returned 17226009 PD 17226010 PD 17226005 R 17226006 R 17226007 R 17226008 R parsl.providers.slurm.slurm DEBUG: Updating job 17226009 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226010 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226008 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:47 proc_lsst.multi:146 [INFO] found job 17226008 in provider checkpoint proc_lsst.multi INFO: found job 17226008 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226005,17226006,17226007,17226008,17226009,17226010' parsl.providers.slurm.slurm DEBUG: squeue returned 17226009 PD 17226010 PD 17226005 R 17226006 R 17226007 R 17226008 R parsl.providers.slurm.slurm DEBUG: Updating job 17226009 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226010 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226008 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:47 proc_lsst.multi:146 [INFO] found job 17226009 in provider checkpoint proc_lsst.multi INFO: found job 17226009 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226005,17226006,17226007,17226008,17226009,17226010' parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99a9b0ddd7a7db5a222069a1560b1051 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/a96d5af6-87a1-49d1-bd8a-43b30042b9b2_isrForCrosstalkSources_1101468_62.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/a96d5af6-87a1-49d1-bd8a-43b30042b9b2_isrForCrosstalkSources_1101468_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17226009 PD 17226010 PD 17226005 R 17226006 R 17226007 R 17226008 R parsl.providers.slurm.slurm DEBUG: Updating job 17226009 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226010 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226008 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:47 proc_lsst.multi:146 [INFO] found job 17226010 in provider checkpoint proc_lsst.multi INFO: found job 17226010 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226005,17226006,17226007,17226008,17226009,17226010' parsl.providers.slurm.slurm DEBUG: squeue returned 17226009 PD 17226010 PD 17226005 R 17226006 R 17226007 R 17226008 R parsl.providers.slurm.slurm DEBUG: Updating job 17226009 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226010 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226008 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 39 parsl.dataflow.strategy DEBUG: Executor multi has 39 active tasks, 60/2 running/pending blocks, and 73 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23453085687824 2024-03-19 00:40:47 proc_lsst.multi:146 [INFO] found job 96275 in provider local proc_lsst.multi INFO: found job 96275 in provider local 2024-03-19 00:40:47 proc_lsst.multi:146 [INFO] found job 17225950 in provider astro proc_lsst.multi INFO: found job 17225950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:47 proc_lsst.multi:146 [INFO] found job 17225951 in provider astro proc_lsst.multi INFO: found job 17225951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:48 proc_lsst.multi:146 [INFO] found job 17225952 in provider astro proc_lsst.multi INFO: found job 17225952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:48 proc_lsst.multi:146 [INFO] found job 17225953 in provider astro proc_lsst.multi INFO: found job 17225953 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:48 proc_lsst.multi:146 [INFO] found job 17225954 in provider astro proc_lsst.multi INFO: found job 17225954 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0068684bad484b17ae0e700655220bf6 with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/92924a35-1721-4bac-b58a-aba06e5d9433_isrForCrosstalkSources_1101465_60.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/92924a35-1721-4bac-b58a-aba06e5d9433_isrForCrosstalkSources_1101465_60.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0c3634aa3a5d09105198068f56c0c97 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/b03b4d62-dfe4-4ca6-93b6-c6c543b9bd06_isrForCrosstalkSources_1101468_57.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/b03b4d62-dfe4-4ca6-93b6-c6c543b9bd06_isrForCrosstalkSources_1101468_57.stderr parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9edcf65488bb43ea38d2b4acc62deb17 with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/1b2aa3ba-af89-4b41-9879-a6b906ea41b5_isrForCrosstalkSources_1101468_39.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/1b2aa3ba-af89-4b41-9879-a6b906ea41b5_isrForCrosstalkSources_1101468_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:49 proc_lsst.multi:146 [INFO] found job 17225955 in provider astro proc_lsst.multi INFO: found job 17225955 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34164c45187d897ce926126c34c36f9a with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/3a01ff1e-83f4-45f5-96c6-19dd99958624_isrForCrosstalkSources_1101468_20.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/3a01ff1e-83f4-45f5-96c6-19dd99958624_isrForCrosstalkSources_1101468_20.stderr parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 618e831a9f83b1c962f43440b4ee3e83 with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/e224e802-0e48-43f9-957a-508a9d420bc6_isrForCrosstalkSources_1101465_32.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/e224e802-0e48-43f9-957a-508a9d420bc6_isrForCrosstalkSources_1101465_32.stderr parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fed8f0d8241451e44b492d7ca54052fc with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/1ecef981-38d7-4275-a920-23f998695a19_isrForCrosstalkSources_1101465_31.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/1ecef981-38d7-4275-a920-23f998695a19_isrForCrosstalkSources_1101465_31.stderr parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3c5f54fbf0e39e7f6675e0649d12354 with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/2ae58f82-10bc-4f35-b013-e1c16c920e0f_isrForCrosstalkSources_1101474_1.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/2ae58f82-10bc-4f35-b013-e1c16c920e0f_isrForCrosstalkSources_1101474_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:49 proc_lsst.multi:146 [INFO] found job 17225956 in provider astro proc_lsst.multi INFO: found job 17225956 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25e48ef147b45d2badbb8ee970609c45 with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/82ddc326-6143-41a4-9ac3-3fd8e6a99087_isrForCrosstalkSources_1101474_17.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/82ddc326-6143-41a4-9ac3-3fd8e6a99087_isrForCrosstalkSources_1101474_17.stderr parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcd19f3798d3fbd35b3f1f963a355c2d with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/6733253b-8d2a-4e79-9f4c-e78a287653f2_isrForCrosstalkSources_1101465_11.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/6733253b-8d2a-4e79-9f4c-e78a287653f2_isrForCrosstalkSources_1101465_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:50 proc_lsst.multi:146 [INFO] found job 17225957 in provider astro proc_lsst.multi INFO: found job 17225957 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c223b57ffce0bd0c37cbed12bbabe38 with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/48bbf511-ac4a-4e59-9465-19407201ab00_isrForCrosstalkSources_1101468_11.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/48bbf511-ac4a-4e59-9465-19407201ab00_isrForCrosstalkSources_1101468_11.stderr parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de2b1ff46bc4634cee32a7b11934f6cd with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/5bb0144c-d0a0-46b7-9c40-986ea7381fff_isrForCrosstalkSources_1101465_36.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/5bb0144c-d0a0-46b7-9c40-986ea7381fff_isrForCrosstalkSources_1101465_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:50 proc_lsst.multi:146 [INFO] found job 17225958 in provider astro proc_lsst.multi INFO: found job 17225958 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4e65b60fb4bae9b377aec17c6f9cb38 with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/c5634bf7-20c2-47a7-8257-eeffa4cabc6a_isrForCrosstalkSources_1101468_40.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/c5634bf7-20c2-47a7-8257-eeffa4cabc6a_isrForCrosstalkSources_1101468_40.stderr parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c6d3d1d64b6bfd224d11231ed84f8d6 with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/bac92ce5-a40b-489b-9e35-7b8dc128fac7_isrForCrosstalkSources_1101474_11.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/bac92ce5-a40b-489b-9e35-7b8dc128fac7_isrForCrosstalkSources_1101474_11.stderr parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b33f8eb81a114d179bb74e68418e60e with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/e5b1c622-0193-4d51-9b3f-1aa6ac09e521_isrForCrosstalkSources_1101474_35.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/e5b1c622-0193-4d51-9b3f-1aa6ac09e521_isrForCrosstalkSources_1101474_35.stderr parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94b664b28dcc0145bc404d91643219c7 with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/12d63abf-e6f0-4f5a-aa91-ae7af215e0f6_isrForCrosstalkSources_1101474_2.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/12d63abf-e6f0-4f5a-aa91-ae7af215e0f6_isrForCrosstalkSources_1101474_2.stderr parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d10c4cfbad9dc8677767a23588f159fd with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/794f8541-aa9a-47c6-bfeb-b0042dcda78b_isrForCrosstalkSources_1101465_2.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/794f8541-aa9a-47c6-bfeb-b0042dcda78b_isrForCrosstalkSources_1101465_2.stderr parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61ae85e8d1204395b52090aaed33aa87 with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/0a92bcbc-bb4e-4f6d-85ba-395319182e7b_isrForCrosstalkSources_1101465_25.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/0a92bcbc-bb4e-4f6d-85ba-395319182e7b_isrForCrosstalkSources_1101465_25.stderr parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc6a12a2109444cb5663430c295af36c with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/d8d4b7c7-6bb2-480c-8342-f33300392d3a_isrForCrosstalkSources_1101468_49.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/d8d4b7c7-6bb2-480c-8342-f33300392d3a_isrForCrosstalkSources_1101468_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:50 proc_lsst.multi:146 [INFO] found job 17225959 in provider astro proc_lsst.multi INFO: found job 17225959 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:51 proc_lsst.multi:146 [INFO] found job 17225960 in provider astro proc_lsst.multi INFO: found job 17225960 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3a47f3e4fb6aa61817d1bfe29338fad with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/4167592d-0cdb-4921-88b4-84b7759a00b8_isrForCrosstalkSources_1101468_48.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/4167592d-0cdb-4921-88b4-84b7759a00b8_isrForCrosstalkSources_1101468_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:51 proc_lsst.multi:146 [INFO] found job 17225961 in provider astro proc_lsst.multi INFO: found job 17225961 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8b4a9fe35a97072f915003e7a59b8aa with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/f316a5e9-3090-4ab4-a155-fc599255d00c_isrForCrosstalkSources_1101471_51.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/f316a5e9-3090-4ab4-a155-fc599255d00c_isrForCrosstalkSources_1101471_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:51 proc_lsst.multi:146 [INFO] found job 17225962 in provider astro proc_lsst.multi INFO: found job 17225962 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c39334c6d9180add580382334ed70e48 with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/b96fb477-7999-4a18-9c3d-d64a7d0b06bf_isrForCrosstalkSources_1101474_27.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/b96fb477-7999-4a18-9c3d-d64a7d0b06bf_isrForCrosstalkSources_1101474_27.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bba11caa3d7d4ec736d45e37537049de with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/837afd65-172e-4849-91ae-fe288674cd78_isrForCrosstalkSources_1101468_29.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/837afd65-172e-4849-91ae-fe288674cd78_isrForCrosstalkSources_1101468_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:51 proc_lsst.multi:146 [INFO] found job 17225963 in provider astro proc_lsst.multi INFO: found job 17225963 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b64320bfa6277664bed32f329f6f8aca with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/20126102-45f1-4025-a1ed-db1e7925ba79_isrForCrosstalkSources_1101471_26.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/20126102-45f1-4025-a1ed-db1e7925ba79_isrForCrosstalkSources_1101471_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:52 proc_lsst.multi:146 [INFO] found job 17225964 in provider astro proc_lsst.multi INFO: found job 17225964 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:52 proc_lsst.multi:146 [INFO] found job 17225965 in provider astro proc_lsst.multi INFO: found job 17225965 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87f74454f8749ba72e37483aea3692fc with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/e47d5e7f-5bd6-4bc4-860f-d55635cabd79_isrForCrosstalkSources_1101471_46.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/e47d5e7f-5bd6-4bc4-860f-d55635cabd79_isrForCrosstalkSources_1101471_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:53 proc_lsst.multi:146 [INFO] found job 17225966 in provider astro proc_lsst.multi INFO: found job 17225966 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 141541d7fbbf7c8b01149232089891d6 with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/3a73c9f9-3e61-47b2-a2cc-2ea6e9ee3a14_isrForCrosstalkSources_1101468_35.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/3a73c9f9-3e61-47b2-a2cc-2ea6e9ee3a14_isrForCrosstalkSources_1101468_35.stderr parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c0429706ce0c92c878283a7584802f5 with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/fdbb5efb-36e4-4d58-8f86-055a5c6182b6_isrForCrosstalkSources_1101468_3.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/fdbb5efb-36e4-4d58-8f86-055a5c6182b6_isrForCrosstalkSources_1101468_3.stderr parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9f5d83aeaddc6b8d258111e4f6fb69c with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/4c17f1aa-eef4-4d8d-bc30-fc3b85432776_isrForCrosstalkSources_1101474_15.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/4c17f1aa-eef4-4d8d-bc30-fc3b85432776_isrForCrosstalkSources_1101474_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:53 proc_lsst.multi:146 [INFO] found job 17225967 in provider astro proc_lsst.multi INFO: found job 17225967 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:54 proc_lsst.multi:146 [INFO] found job 17225968 in provider astro proc_lsst.multi INFO: found job 17225968 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:54 proc_lsst.multi:146 [INFO] found job 17225969 in provider astro proc_lsst.multi INFO: found job 17225969 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:55 proc_lsst.multi:146 [INFO] found job 17225970 in provider astro proc_lsst.multi INFO: found job 17225970 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:55 proc_lsst.multi:146 [INFO] found job 17225971 in provider astro proc_lsst.multi INFO: found job 17225971 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c163dd21325c36a0f8b04b3b9836713 with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/4a224eea-00f7-4771-b201-2c855579de51_isrForCrosstalkSources_1101474_33.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/4a224eea-00f7-4771-b201-2c855579de51_isrForCrosstalkSources_1101474_33.stderr parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbf89e4d9863615c4e78ae63c2d4427f with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/aa33965d-da90-4669-af74-ee9bc8017873_isrForCrosstalkSources_1101465_9.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/aa33965d-da90-4669-af74-ee9bc8017873_isrForCrosstalkSources_1101465_9.stderr parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 981b287b242e6e1593277e41c8564d72 with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/85f0a2b5-c86e-44d2-a636-abf8f849b140_isrForCrosstalkSources_1101471_19.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/85f0a2b5-c86e-44d2-a636-abf8f849b140_isrForCrosstalkSources_1101471_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:55 proc_lsst.multi:146 [INFO] found job 17225972 in provider astro proc_lsst.multi INFO: found job 17225972 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b4f62a75fbce0535061ededb6b00ead with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/f099f506-bfd5-4586-9f15-a2496b63857f_isrForCrosstalkSources_1101465_47.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/f099f506-bfd5-4586-9f15-a2496b63857f_isrForCrosstalkSources_1101465_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:56 proc_lsst.multi:146 [INFO] found job 17225973 in provider astro proc_lsst.multi INFO: found job 17225973 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 581df70bb61afe4abd5eda55ec115240 with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/91042eb3-207b-4f06-964d-ed17484b5fa1_isrForCrosstalkSources_1101471_49.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/91042eb3-207b-4f06-964d-ed17484b5fa1_isrForCrosstalkSources_1101471_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:56 proc_lsst.multi:146 [INFO] found job 17225974 in provider astro proc_lsst.multi INFO: found job 17225974 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a7bff1189eecb0a51182177f1662892 with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/585d8c72-ebde-41fa-9f73-76a7ee48ae0c_isrForCrosstalkSources_1101471_5.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/585d8c72-ebde-41fa-9f73-76a7ee48ae0c_isrForCrosstalkSources_1101471_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:56 proc_lsst.multi:146 [INFO] found job 17225975 in provider astro proc_lsst.multi INFO: found job 17225975 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:57 proc_lsst.multi:146 [INFO] found job 17225976 in provider astro proc_lsst.multi INFO: found job 17225976 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0536a90340b21297061682fab87ba65 with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/38ec3768-d768-42c8-aacb-e59542af779e_isrForCrosstalkSources_1101471_42.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/38ec3768-d768-42c8-aacb-e59542af779e_isrForCrosstalkSources_1101471_42.stderr parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1a3b01da470b40ae6f8e6cb1afa2696 with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/3f244fff-8b4a-4678-8121-1556d19f6429_isrForCrosstalkSources_1101474_52.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101474/3f244fff-8b4a-4678-8121-1556d19f6429_isrForCrosstalkSources_1101474_52.stderr parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0840a28b3ca5e1460f20f2c1dcaa511a with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/5ca15d87-8508-470a-9288-460e25ccb988_isrForCrosstalkSources_1101468_12.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/5ca15d87-8508-470a-9288-460e25ccb988_isrForCrosstalkSources_1101468_12.stderr parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afe7ad161c7ab7b0cbfb2cbefef43127 with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/4b86af2d-b7a9-4a19-870a-a9847d64af10_isrForCrosstalkSources_1101471_43.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101471/4b86af2d-b7a9-4a19-870a-a9847d64af10_isrForCrosstalkSources_1101471_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:57 proc_lsst.multi:146 [INFO] found job 17225977 in provider astro proc_lsst.multi INFO: found job 17225977 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:57 proc_lsst.multi:146 [INFO] found job 17225978 in provider astro proc_lsst.multi INFO: found job 17225978 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:57 proc_lsst.multi:146 [INFO] found job 17225979 in provider astro proc_lsst.multi INFO: found job 17225979 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225950,17225951,17225952,17225953,17225954,17225955,17225956,17225957,17225958,17225959,17225960,17225961,17225962,17225963,17225964,17225965,17225966,17225967,17225968,17225969,17225970,17225971,17225972,17225973,17225974,17225975,17225976,17225977,17225978,17225979' parsl.providers.slurm.slurm DEBUG: squeue returned 17225978 R 17225979 R 17225973 R 17225974 R 17225975 R 17225976 R 17225977 R 17225972 R 17225968 R 17225969 R 17225970 R 17225971 R 17225965 R 17225966 R 17225967 R 17225964 R 17225956 R 17225957 R 17225958 R 17225959 R 17225960 R 17225961 R 17225962 R 17225963 R 17225952 R 17225953 R 17225954 R 17225955 R 17225950 R 17225951 R parsl.providers.slurm.slurm DEBUG: Updating job 17225978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225951 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:57 proc_lsst.multi:146 [INFO] found job 17225980 in provider escience_rtx6k proc_lsst.multi INFO: found job 17225980 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225980,17225981,17225982,17225983,17225984' parsl.providers.slurm.slurm DEBUG: squeue returned 17225983 R 17225984 R 17225981 R 17225982 R 17225980 R parsl.providers.slurm.slurm DEBUG: Updating job 17225983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225980 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:57 proc_lsst.multi:146 [INFO] found job 17225981 in provider escience_rtx6k proc_lsst.multi INFO: found job 17225981 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225980,17225981,17225982,17225983,17225984' parsl.providers.slurm.slurm DEBUG: squeue returned 17225983 R 17225984 R 17225981 R 17225982 R 17225980 R parsl.providers.slurm.slurm DEBUG: Updating job 17225983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225980 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:58 proc_lsst.multi:146 [INFO] found job 17225982 in provider escience_rtx6k proc_lsst.multi INFO: found job 17225982 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225980,17225981,17225982,17225983,17225984' parsl.providers.slurm.slurm DEBUG: squeue returned 17225983 R 17225984 R 17225981 R 17225982 R 17225980 R parsl.providers.slurm.slurm DEBUG: Updating job 17225983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225980 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:58 proc_lsst.multi:146 [INFO] found job 17225983 in provider escience_rtx6k proc_lsst.multi INFO: found job 17225983 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225980,17225981,17225982,17225983,17225984' parsl.providers.slurm.slurm DEBUG: squeue returned 17225983 R 17225984 R 17225981 R 17225982 R 17225980 R parsl.providers.slurm.slurm DEBUG: Updating job 17225983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225980 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:58 proc_lsst.multi:146 [INFO] found job 17225984 in provider escience_rtx6k proc_lsst.multi INFO: found job 17225984 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225980,17225981,17225982,17225983,17225984' parsl.providers.slurm.slurm DEBUG: squeue returned 17225983 R 17225984 R 17225981 R 17225982 R 17225980 R parsl.providers.slurm.slurm DEBUG: Updating job 17225983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225980 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:58 proc_lsst.multi:146 [INFO] found job 17225985 in provider escience_a40 proc_lsst.multi INFO: found job 17225985 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:58 proc_lsst.multi:146 [INFO] found job 17225986 in provider escience_a40 proc_lsst.multi INFO: found job 17225986 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:58 proc_lsst.multi:146 [INFO] found job 17225987 in provider escience_a40 proc_lsst.multi INFO: found job 17225987 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:59 proc_lsst.multi:146 [INFO] found job 17225988 in provider escience_a40 proc_lsst.multi INFO: found job 17225988 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:59 proc_lsst.multi:146 [INFO] found job 17225989 in provider escience_a40 proc_lsst.multi INFO: found job 17225989 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:59 proc_lsst.multi:146 [INFO] found job 17225990 in provider escience_a40 proc_lsst.multi INFO: found job 17225990 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:59 proc_lsst.multi:146 [INFO] found job 17225991 in provider escience_a40 proc_lsst.multi INFO: found job 17225991 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:59 proc_lsst.multi:146 [INFO] found job 17225992 in provider escience_a40 proc_lsst.multi INFO: found job 17225992 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:40:59 proc_lsst.multi:146 [INFO] found job 17225993 in provider escience_a40 proc_lsst.multi INFO: found job 17225993 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:41:00 proc_lsst.multi:146 [INFO] found job 17225994 in provider escience_a40 proc_lsst.multi INFO: found job 17225994 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:41:00 proc_lsst.multi:146 [INFO] found job 17225995 in provider escience_a40 proc_lsst.multi INFO: found job 17225995 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:41:01 proc_lsst.multi:146 [INFO] found job 17225996 in provider escience_a40 proc_lsst.multi INFO: found job 17225996 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:41:01 proc_lsst.multi:146 [INFO] found job 17225997 in provider escience_a40 proc_lsst.multi INFO: found job 17225997 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:41:01 proc_lsst.multi:146 [INFO] found job 17225998 in provider escience_a40 proc_lsst.multi INFO: found job 17225998 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:41:01 proc_lsst.multi:146 [INFO] found job 17225999 in provider escience_a40 proc_lsst.multi INFO: found job 17225999 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:41:01 proc_lsst.multi:146 [INFO] found job 17226000 in provider escience_a40 proc_lsst.multi INFO: found job 17226000 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:41:01 proc_lsst.multi:146 [INFO] found job 17226001 in provider escience_a40 proc_lsst.multi INFO: found job 17226001 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:41:02 proc_lsst.multi:146 [INFO] found job 17226002 in provider escience_a40 proc_lsst.multi INFO: found job 17226002 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:41:02 proc_lsst.multi:146 [INFO] found job 17226003 in provider escience_a40 proc_lsst.multi INFO: found job 17226003 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:41:02 proc_lsst.multi:146 [INFO] found job 17226004 in provider escience_a40 proc_lsst.multi INFO: found job 17226004 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17225985,17225986,17225987,17225988,17225989,17225990,17225991,17225992,17225993,17225994,17225995,17225996,17225997,17225998,17225999,17226000,17226001,17226002,17226003,17226004' parsl.providers.slurm.slurm DEBUG: squeue returned 17226004 R 17226003 R 17226002 R 17225999 R 17226000 R 17226001 R 17225997 R 17225998 R 17225996 R 17225992 R 17225993 R 17225994 R 17225995 R 17225990 R 17225991 R 17225986 R 17225987 R 17225988 R 17225989 R 17225985 R parsl.providers.slurm.slurm DEBUG: Updating job 17226004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17225985 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:41:02 proc_lsst.multi:146 [INFO] found job 17226005 in provider checkpoint proc_lsst.multi INFO: found job 17226005 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226005,17226006,17226007,17226008,17226009,17226010' parsl.providers.slurm.slurm DEBUG: squeue returned 17226005 CG 17226007 CG 17226009 PD 17226010 PD 17226006 R 17226008 R parsl.providers.slurm.slurm DEBUG: Updating job 17226005 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226007 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226009 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226010 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226008 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:41:02 proc_lsst.multi:146 [INFO] found job 17226006 in provider checkpoint proc_lsst.multi INFO: found job 17226006 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226005,17226006,17226007,17226008,17226009,17226010' parsl.providers.slurm.slurm DEBUG: squeue returned 17226005 CG 17226007 CG 17226009 PD 17226010 PD 17226006 R 17226008 R parsl.providers.slurm.slurm DEBUG: Updating job 17226005 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226007 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226009 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226010 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226008 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:41:02 proc_lsst.multi:146 [INFO] found job 17226007 in provider checkpoint proc_lsst.multi INFO: found job 17226007 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226005,17226006,17226007,17226008,17226009,17226010' parsl.providers.slurm.slurm DEBUG: squeue returned 17226005 CG 17226007 CG 17226009 PD 17226010 PD 17226006 R 17226008 R parsl.providers.slurm.slurm DEBUG: Updating job 17226005 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226007 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226009 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226010 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226008 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:41:02 proc_lsst.multi:146 [INFO] found job 17226008 in provider checkpoint proc_lsst.multi INFO: found job 17226008 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226005,17226006,17226007,17226008,17226009,17226010' parsl.providers.slurm.slurm DEBUG: squeue returned 17226005 CG 17226007 CG 17226009 PD 17226010 PD 17226006 R 17226008 R parsl.providers.slurm.slurm DEBUG: Updating job 17226005 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226007 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226009 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226010 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226008 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:41:03 proc_lsst.multi:146 [INFO] found job 17226009 in provider checkpoint proc_lsst.multi INFO: found job 17226009 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226005,17226006,17226007,17226008,17226009,17226010' parsl.providers.slurm.slurm DEBUG: squeue returned 17226005 CG 17226007 CG 17226009 PD 17226010 PD 17226006 R 17226008 R parsl.providers.slurm.slurm DEBUG: Updating job 17226005 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226007 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226009 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226010 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226008 with slurm status R to parsl state JobState.RUNNING 2024-03-19 00:41:03 proc_lsst.multi:146 [INFO] found job 17226010 in provider checkpoint proc_lsst.multi INFO: found job 17226010 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17226005,17226006,17226007,17226008,17226009,17226010' parsl.providers.slurm.slurm DEBUG: squeue returned 17226005 CG 17226007 CG 17226009 PD 17226010 PD 17226006 R 17226008 R parsl.providers.slurm.slurm DEBUG: Updating job 17226005 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226007 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226009 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226010 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17226006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17226008 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23453085687824 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23453085687824 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23453085687824 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23453085687824 parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3541c8a365619b8d3de629ea5d1e428 with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/19cf25d0-c3c5-4de0-8cfd-8def429126f4_isrForCrosstalkSources_1101468_17.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101468/19cf25d0-c3c5-4de0-8cfd-8def429126f4_isrForCrosstalkSources_1101468_17.stderr parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69d2a98de29219137000d3cd369f39c1 with result from task 247 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 248 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Tasks in state States.failed: 0 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/90281749-4ca9-41df-b04a-38b245eb86de_isrForCrosstalkSources_1101465_12.stdout parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/submit/DEEP/20220527/A0b/science#step0/20240319T003842Z/logs/isrForCrosstalkSources/1101465/90281749-4ca9-41df-b04a-38b245eb86de_isrForCrosstalkSources_1101465_12.stderr parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=62, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 60 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c84dcc55ba81 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c84dcc55ba81 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ad20196b9305 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ad20196b9305 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a7babbfb03df parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a7babbfb03df parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: aab4e17088a8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: aab4e17088a8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d903bfade211 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d903bfade211 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d2a1651d4053 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d2a1651d4053 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2d4dd65bb913 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2d4dd65bb913 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 962942a3fd3c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 962942a3fd3c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bf57ee7d7362 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bf57ee7d7362 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5368e3236fc9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5368e3236fc9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c39768303bb5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c39768303bb5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 376116b2823d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 376116b2823d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ccb7bd6d77d2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ccb7bd6d77d2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b3335e2e5d7d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b3335e2e5d7d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3e1d83a8bc15 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3e1d83a8bc15 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 51674ca98764 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 51674ca98764 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9d86ca5bf6b1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9d86ca5bf6b1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 97a8f777819c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 97a8f777819c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0783126b22cd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0783126b22cd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8f6526af62a0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8f6526af62a0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 87ec237bce24 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 87ec237bce24 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4a1d485c40c3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4a1d485c40c3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7f14b765c3f6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7f14b765c3f6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b61b457ffac9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b61b457ffac9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 41af867c83d3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 41af867c83d3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 75f36956c86c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 75f36956c86c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2460ecef5588 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2460ecef5588 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 634635f8c484 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 634635f8c484 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 848120091073 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 848120091073 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dcd44cd1d53c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dcd44cd1d53c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1c0cb82a5ee1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1c0cb82a5ee1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1cda8fab6453 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1cda8fab6453 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0be6520f81df parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0be6520f81df parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6a6fe2a8bc0b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6a6fe2a8bc0b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dfbecc66f707 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dfbecc66f707 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 261380ae0db5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 261380ae0db5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 52f51b8e7999 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 52f51b8e7999 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 460f0723f40e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 460f0723f40e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4da84d5ecd15 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4da84d5ecd15 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5db9b9ab585b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5db9b9ab585b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2a64645848c5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2a64645848c5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0ae582855409 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0ae582855409 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ec47874a4554 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ec47874a4554 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b199bc0d05e5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b199bc0d05e5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b57f5a805a06 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b57f5a805a06 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 899f1ee84ce3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 899f1ee84ce3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3cc9f3c01616 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3cc9f3c01616 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 56289021722f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 56289021722f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7d300bc215d2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7d300bc215d2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6a363f61fea5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6a363f61fea5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9bd423ab1590 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9bd423ab1590 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 163b17740a3c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 163b17740a3c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2b77e9b962db parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2b77e9b962db parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1172974547c9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1172974547c9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 226f0bdf13c1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 226f0bdf13c1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fd4a2ed5ff59 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fd4a2ed5ff59 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dbf98f33e5a7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dbf98f33e5a7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6419e1c38584 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6419e1c38584 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 98bf3f1f9d8b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 98bf3f1f9d8b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f9220a36b74b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f9220a36b74b 2024-03-19 00:41:16 proc_lsst.multi:146 [INFO] found job 17226006 in provider checkpoint proc_lsst.multi INFO: found job 17226006 in provider checkpoint 2024-03-19 00:41:16 proc_lsst.multi:201 [INFO] cancelling 17226006 on provider checkpoint proc_lsst.multi INFO: cancelling 17226006 on provider checkpoint 2024-03-19 00:41:17 proc_lsst.multi:146 [INFO] found job 17225969 in provider astro proc_lsst.multi INFO: found job 17225969 in provider astro 2024-03-19 00:41:17 proc_lsst.multi:201 [INFO] cancelling 17225969 on provider astro proc_lsst.multi INFO: cancelling 17225969 on provider astro 2024-03-19 00:41:17 proc_lsst.multi:146 [INFO] found job 17225978 in provider astro proc_lsst.multi INFO: found job 17225978 in provider astro 2024-03-19 00:41:17 proc_lsst.multi:201 [INFO] cancelling 17225978 on provider astro proc_lsst.multi INFO: cancelling 17225978 on provider astro 2024-03-19 00:41:17 proc_lsst.multi:146 [INFO] found job 17225960 in provider astro proc_lsst.multi INFO: found job 17225960 in provider astro 2024-03-19 00:41:17 proc_lsst.multi:201 [INFO] cancelling 17225960 on provider astro proc_lsst.multi INFO: cancelling 17225960 on provider astro 2024-03-19 00:41:17 proc_lsst.multi:146 [INFO] found job 17225961 in provider astro proc_lsst.multi INFO: found job 17225961 in provider astro 2024-03-19 00:41:17 proc_lsst.multi:201 [INFO] cancelling 17225961 on provider astro proc_lsst.multi INFO: cancelling 17225961 on provider astro 2024-03-19 00:41:17 proc_lsst.multi:146 [INFO] found job 17225964 in provider astro proc_lsst.multi INFO: found job 17225964 in provider astro 2024-03-19 00:41:17 proc_lsst.multi:201 [INFO] cancelling 17225964 on provider astro proc_lsst.multi INFO: cancelling 17225964 on provider astro 2024-03-19 00:41:17 proc_lsst.multi:146 [INFO] found job 96275 in provider local proc_lsst.multi INFO: found job 96275 in provider local 2024-03-19 00:41:17 proc_lsst.multi:201 [INFO] cancelling 96275 on provider local proc_lsst.multi INFO: cancelling 96275 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 96275 2024-03-19 00:41:17 proc_lsst.multi:146 [INFO] found job 17225974 in provider astro proc_lsst.multi INFO: found job 17225974 in provider astro 2024-03-19 00:41:17 proc_lsst.multi:201 [INFO] cancelling 17225974 on provider astro proc_lsst.multi INFO: cancelling 17225974 on provider astro 2024-03-19 00:41:17 proc_lsst.multi:146 [INFO] found job 17225967 in provider astro proc_lsst.multi INFO: found job 17225967 in provider astro 2024-03-19 00:41:17 proc_lsst.multi:201 [INFO] cancelling 17225967 on provider astro proc_lsst.multi INFO: cancelling 17225967 on provider astro 2024-03-19 00:41:17 proc_lsst.multi:146 [INFO] found job 17225962 in provider astro proc_lsst.multi INFO: found job 17225962 in provider astro 2024-03-19 00:41:17 proc_lsst.multi:201 [INFO] cancelling 17225962 on provider astro proc_lsst.multi INFO: cancelling 17225962 on provider astro 2024-03-19 00:41:18 proc_lsst.multi:146 [INFO] found job 17225965 in provider astro proc_lsst.multi INFO: found job 17225965 in provider astro 2024-03-19 00:41:18 proc_lsst.multi:201 [INFO] cancelling 17225965 on provider astro proc_lsst.multi INFO: cancelling 17225965 on provider astro 2024-03-19 00:41:18 proc_lsst.multi:146 [INFO] found job 17225951 in provider astro proc_lsst.multi INFO: found job 17225951 in provider astro 2024-03-19 00:41:18 proc_lsst.multi:201 [INFO] cancelling 17225951 on provider astro proc_lsst.multi INFO: cancelling 17225951 on provider astro 2024-03-19 00:41:18 proc_lsst.multi:146 [INFO] found job 17225958 in provider astro proc_lsst.multi INFO: found job 17225958 in provider astro 2024-03-19 00:41:18 proc_lsst.multi:201 [INFO] cancelling 17225958 on provider astro proc_lsst.multi INFO: cancelling 17225958 on provider astro 2024-03-19 00:41:18 proc_lsst.multi:146 [INFO] found job 17225957 in provider astro proc_lsst.multi INFO: found job 17225957 in provider astro 2024-03-19 00:41:18 proc_lsst.multi:201 [INFO] cancelling 17225957 on provider astro proc_lsst.multi INFO: cancelling 17225957 on provider astro 2024-03-19 00:41:18 proc_lsst.multi:146 [INFO] found job 17225956 in provider astro proc_lsst.multi INFO: found job 17225956 in provider astro 2024-03-19 00:41:18 proc_lsst.multi:201 [INFO] cancelling 17225956 on provider astro proc_lsst.multi INFO: cancelling 17225956 on provider astro 2024-03-19 00:41:18 proc_lsst.multi:146 [INFO] found job 17225955 in provider astro proc_lsst.multi INFO: found job 17225955 in provider astro 2024-03-19 00:41:18 proc_lsst.multi:201 [INFO] cancelling 17225955 on provider astro proc_lsst.multi INFO: cancelling 17225955 on provider astro 2024-03-19 00:41:18 proc_lsst.multi:146 [INFO] found job 17225954 in provider astro proc_lsst.multi INFO: found job 17225954 in provider astro 2024-03-19 00:41:18 proc_lsst.multi:201 [INFO] cancelling 17225954 on provider astro proc_lsst.multi INFO: cancelling 17225954 on provider astro 2024-03-19 00:41:18 proc_lsst.multi:146 [INFO] found job 17225959 in provider astro proc_lsst.multi INFO: found job 17225959 in provider astro 2024-03-19 00:41:18 proc_lsst.multi:201 [INFO] cancelling 17225959 on provider astro proc_lsst.multi INFO: cancelling 17225959 on provider astro 2024-03-19 00:41:18 proc_lsst.multi:146 [INFO] found job 17225994 in provider escience_a40 proc_lsst.multi INFO: found job 17225994 in provider escience_a40 2024-03-19 00:41:18 proc_lsst.multi:201 [INFO] cancelling 17225994 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225994 on provider escience_a40 2024-03-19 00:41:19 proc_lsst.multi:146 [INFO] found job 17225988 in provider escience_a40 proc_lsst.multi INFO: found job 17225988 in provider escience_a40 2024-03-19 00:41:19 proc_lsst.multi:201 [INFO] cancelling 17225988 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225988 on provider escience_a40 2024-03-19 00:41:19 proc_lsst.multi:146 [INFO] found job 17225990 in provider escience_a40 proc_lsst.multi INFO: found job 17225990 in provider escience_a40 2024-03-19 00:41:19 proc_lsst.multi:201 [INFO] cancelling 17225990 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225990 on provider escience_a40 2024-03-19 00:41:19 proc_lsst.multi:146 [INFO] found job 17225995 in provider escience_a40 proc_lsst.multi INFO: found job 17225995 in provider escience_a40 2024-03-19 00:41:19 proc_lsst.multi:201 [INFO] cancelling 17225995 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225995 on provider escience_a40 2024-03-19 00:41:19 proc_lsst.multi:146 [INFO] found job 17226000 in provider escience_a40 proc_lsst.multi INFO: found job 17226000 in provider escience_a40 2024-03-19 00:41:19 proc_lsst.multi:201 [INFO] cancelling 17226000 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226000 on provider escience_a40 2024-03-19 00:41:19 proc_lsst.multi:146 [INFO] found job 17226005 in provider checkpoint proc_lsst.multi INFO: found job 17226005 in provider checkpoint 2024-03-19 00:41:19 proc_lsst.multi:201 [INFO] cancelling 17226005 on provider checkpoint proc_lsst.multi INFO: cancelling 17226005 on provider checkpoint 2024-03-19 00:41:19 proc_lsst.multi:146 [INFO] found job 17225991 in provider escience_a40 proc_lsst.multi INFO: found job 17225991 in provider escience_a40 2024-03-19 00:41:19 proc_lsst.multi:201 [INFO] cancelling 17225991 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225991 on provider escience_a40 2024-03-19 00:41:19 proc_lsst.multi:146 [INFO] found job 17225997 in provider escience_a40 proc_lsst.multi INFO: found job 17225997 in provider escience_a40 2024-03-19 00:41:19 proc_lsst.multi:201 [INFO] cancelling 17225997 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225997 on provider escience_a40 2024-03-19 00:41:19 proc_lsst.multi:146 [INFO] found job 17226001 in provider escience_a40 proc_lsst.multi INFO: found job 17226001 in provider escience_a40 2024-03-19 00:41:19 proc_lsst.multi:201 [INFO] cancelling 17226001 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226001 on provider escience_a40 2024-03-19 00:41:19 proc_lsst.multi:146 [INFO] found job 17225996 in provider escience_a40 proc_lsst.multi INFO: found job 17225996 in provider escience_a40 2024-03-19 00:41:19 proc_lsst.multi:201 [INFO] cancelling 17225996 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225996 on provider escience_a40 2024-03-19 00:41:20 proc_lsst.multi:146 [INFO] found job 17225989 in provider escience_a40 proc_lsst.multi INFO: found job 17225989 in provider escience_a40 2024-03-19 00:41:20 proc_lsst.multi:201 [INFO] cancelling 17225989 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225989 on provider escience_a40 2024-03-19 00:41:20 proc_lsst.multi:146 [INFO] found job 17225993 in provider escience_a40 proc_lsst.multi INFO: found job 17225993 in provider escience_a40 2024-03-19 00:41:20 proc_lsst.multi:201 [INFO] cancelling 17225993 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225993 on provider escience_a40 2024-03-19 00:41:20 proc_lsst.multi:146 [INFO] found job 17225999 in provider escience_a40 proc_lsst.multi INFO: found job 17225999 in provider escience_a40 2024-03-19 00:41:20 proc_lsst.multi:201 [INFO] cancelling 17225999 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225999 on provider escience_a40 2024-03-19 00:41:20 proc_lsst.multi:146 [INFO] found job 17226008 in provider checkpoint proc_lsst.multi INFO: found job 17226008 in provider checkpoint 2024-03-19 00:41:20 proc_lsst.multi:201 [INFO] cancelling 17226008 on provider checkpoint proc_lsst.multi INFO: cancelling 17226008 on provider checkpoint 2024-03-19 00:41:20 proc_lsst.multi:146 [INFO] found job 17225987 in provider escience_a40 proc_lsst.multi INFO: found job 17225987 in provider escience_a40 2024-03-19 00:41:20 proc_lsst.multi:201 [INFO] cancelling 17225987 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225987 on provider escience_a40 2024-03-19 00:41:20 proc_lsst.multi:146 [INFO] found job 17226003 in provider escience_a40 proc_lsst.multi INFO: found job 17226003 in provider escience_a40 2024-03-19 00:41:20 proc_lsst.multi:201 [INFO] cancelling 17226003 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226003 on provider escience_a40 2024-03-19 00:41:20 proc_lsst.multi:146 [INFO] found job 17225992 in provider escience_a40 proc_lsst.multi INFO: found job 17225992 in provider escience_a40 2024-03-19 00:41:20 proc_lsst.multi:201 [INFO] cancelling 17225992 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225992 on provider escience_a40 2024-03-19 00:41:20 proc_lsst.multi:146 [INFO] found job 17225998 in provider escience_a40 proc_lsst.multi INFO: found job 17225998 in provider escience_a40 2024-03-19 00:41:20 proc_lsst.multi:201 [INFO] cancelling 17225998 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225998 on provider escience_a40 2024-03-19 00:41:20 proc_lsst.multi:146 [INFO] found job 17225986 in provider escience_a40 proc_lsst.multi INFO: found job 17225986 in provider escience_a40 2024-03-19 00:41:20 proc_lsst.multi:201 [INFO] cancelling 17225986 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225986 on provider escience_a40 2024-03-19 00:41:21 proc_lsst.multi:146 [INFO] found job 17226004 in provider escience_a40 proc_lsst.multi INFO: found job 17226004 in provider escience_a40 2024-03-19 00:41:21 proc_lsst.multi:201 [INFO] cancelling 17226004 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226004 on provider escience_a40 2024-03-19 00:41:21 proc_lsst.multi:146 [INFO] found job 17225984 in provider escience_rtx6k proc_lsst.multi INFO: found job 17225984 in provider escience_rtx6k 2024-03-19 00:41:21 proc_lsst.multi:201 [INFO] cancelling 17225984 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17225984 on provider escience_rtx6k 2024-03-19 00:41:21 proc_lsst.multi:146 [INFO] found job 17226002 in provider escience_a40 proc_lsst.multi INFO: found job 17226002 in provider escience_a40 2024-03-19 00:41:21 proc_lsst.multi:201 [INFO] cancelling 17226002 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226002 on provider escience_a40 2024-03-19 00:41:21 proc_lsst.multi:146 [INFO] found job 17225982 in provider escience_rtx6k proc_lsst.multi INFO: found job 17225982 in provider escience_rtx6k 2024-03-19 00:41:21 proc_lsst.multi:201 [INFO] cancelling 17225982 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17225982 on provider escience_rtx6k 2024-03-19 00:41:21 proc_lsst.multi:146 [INFO] found job 17225952 in provider astro proc_lsst.multi INFO: found job 17225952 in provider astro 2024-03-19 00:41:21 proc_lsst.multi:201 [INFO] cancelling 17225952 on provider astro proc_lsst.multi INFO: cancelling 17225952 on provider astro 2024-03-19 00:41:21 proc_lsst.multi:146 [INFO] found job 17225973 in provider astro proc_lsst.multi INFO: found job 17225973 in provider astro 2024-03-19 00:41:21 proc_lsst.multi:201 [INFO] cancelling 17225973 on provider astro proc_lsst.multi INFO: cancelling 17225973 on provider astro 2024-03-19 00:41:21 proc_lsst.multi:146 [INFO] found job 17225953 in provider astro proc_lsst.multi INFO: found job 17225953 in provider astro 2024-03-19 00:41:21 proc_lsst.multi:201 [INFO] cancelling 17225953 on provider astro proc_lsst.multi INFO: cancelling 17225953 on provider astro 2024-03-19 00:41:21 proc_lsst.multi:146 [INFO] found job 17225981 in provider escience_rtx6k proc_lsst.multi INFO: found job 17225981 in provider escience_rtx6k 2024-03-19 00:41:21 proc_lsst.multi:201 [INFO] cancelling 17225981 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17225981 on provider escience_rtx6k 2024-03-19 00:41:21 proc_lsst.multi:146 [INFO] found job 17225980 in provider escience_rtx6k proc_lsst.multi INFO: found job 17225980 in provider escience_rtx6k 2024-03-19 00:41:21 proc_lsst.multi:201 [INFO] cancelling 17225980 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17225980 on provider escience_rtx6k 2024-03-19 00:41:22 proc_lsst.multi:146 [INFO] found job 17225983 in provider escience_rtx6k proc_lsst.multi INFO: found job 17225983 in provider escience_rtx6k 2024-03-19 00:41:22 proc_lsst.multi:201 [INFO] cancelling 17225983 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17225983 on provider escience_rtx6k 2024-03-19 00:41:22 proc_lsst.multi:146 [INFO] found job 17225977 in provider astro proc_lsst.multi INFO: found job 17225977 in provider astro 2024-03-19 00:41:22 proc_lsst.multi:201 [INFO] cancelling 17225977 on provider astro proc_lsst.multi INFO: cancelling 17225977 on provider astro 2024-03-19 00:41:22 proc_lsst.multi:146 [INFO] found job 17225968 in provider astro proc_lsst.multi INFO: found job 17225968 in provider astro 2024-03-19 00:41:22 proc_lsst.multi:201 [INFO] cancelling 17225968 on provider astro proc_lsst.multi INFO: cancelling 17225968 on provider astro 2024-03-19 00:41:22 proc_lsst.multi:146 [INFO] found job 17225985 in provider escience_a40 proc_lsst.multi INFO: found job 17225985 in provider escience_a40 2024-03-19 00:41:22 proc_lsst.multi:201 [INFO] cancelling 17225985 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225985 on provider escience_a40 2024-03-19 00:41:22 proc_lsst.multi:146 [INFO] found job 17225972 in provider astro proc_lsst.multi INFO: found job 17225972 in provider astro 2024-03-19 00:41:22 proc_lsst.multi:201 [INFO] cancelling 17225972 on provider astro proc_lsst.multi INFO: cancelling 17225972 on provider astro 2024-03-19 00:41:22 proc_lsst.multi:146 [INFO] found job 17225979 in provider astro proc_lsst.multi INFO: found job 17225979 in provider astro 2024-03-19 00:41:22 proc_lsst.multi:201 [INFO] cancelling 17225979 on provider astro proc_lsst.multi INFO: cancelling 17225979 on provider astro 2024-03-19 00:41:22 proc_lsst.multi:146 [INFO] found job 17225970 in provider astro proc_lsst.multi INFO: found job 17225970 in provider astro 2024-03-19 00:41:22 proc_lsst.multi:201 [INFO] cancelling 17225970 on provider astro proc_lsst.multi INFO: cancelling 17225970 on provider astro 2024-03-19 00:41:22 proc_lsst.multi:146 [INFO] found job 17225966 in provider astro proc_lsst.multi INFO: found job 17225966 in provider astro 2024-03-19 00:41:22 proc_lsst.multi:201 [INFO] cancelling 17225966 on provider astro proc_lsst.multi INFO: cancelling 17225966 on provider astro 2024-03-19 00:41:22 proc_lsst.multi:146 [INFO] found job 17225963 in provider astro proc_lsst.multi INFO: found job 17225963 in provider astro 2024-03-19 00:41:22 proc_lsst.multi:201 [INFO] cancelling 17225963 on provider astro proc_lsst.multi INFO: cancelling 17225963 on provider astro 2024-03-19 00:41:23 proc_lsst.multi:146 [INFO] found job 17226007 in provider checkpoint proc_lsst.multi INFO: found job 17226007 in provider checkpoint 2024-03-19 00:41:23 proc_lsst.multi:201 [INFO] cancelling 17226007 on provider checkpoint proc_lsst.multi INFO: cancelling 17226007 on provider checkpoint 2024-03-19 00:41:23 proc_lsst.multi:146 [INFO] found job 17225950 in provider astro proc_lsst.multi INFO: found job 17225950 in provider astro 2024-03-19 00:41:23 proc_lsst.multi:201 [INFO] cancelling 17225950 on provider astro proc_lsst.multi INFO: cancelling 17225950 on provider astro 2024-03-19 00:41:23 proc_lsst.multi:146 [INFO] found job 17225971 in provider astro proc_lsst.multi INFO: found job 17225971 in provider astro 2024-03-19 00:41:23 proc_lsst.multi:201 [INFO] cancelling 17225971 on provider astro proc_lsst.multi INFO: cancelling 17225971 on provider astro 2024-03-19 00:41:23 proc_lsst.multi:146 [INFO] found job 17225975 in provider astro proc_lsst.multi INFO: found job 17225975 in provider astro 2024-03-19 00:41:23 proc_lsst.multi:201 [INFO] cancelling 17225975 on provider astro proc_lsst.multi INFO: cancelling 17225975 on provider astro 2024-03-19 00:41:23 proc_lsst.multi:146 [INFO] found job 17225976 in provider astro proc_lsst.multi INFO: found job 17225976 in provider astro 2024-03-19 00:41:23 proc_lsst.multi:201 [INFO] cancelling 17225976 on provider astro proc_lsst.multi INFO: cancelling 17225976 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-19 00:41:23 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-19 00:41:23 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17226005', '17226006', '17226007', '17226008', '17226009', '17226010', '17225950', '17225951', '17225952', '17225953', '17225954', '17225955', '17225956', '17225957', '17225958', '17225959', '17225960', '17225961', '17225962', '17225963', '17225964', '17225965', '17225966', '17225967', '17225968', '17225969', '17225970', '17225971', '17225972', '17225973', '17225974', '17225975', '17225976', '17225977', '17225978', '17225979', '17225985', '17225986', '17225987', '17225988', '17225989', '17225990', '17225991', '17225992', '17225993', '17225994', '17225995', '17225996', '17225997', '17225998', '17225999', '17226000', '17226001', '17226002', '17226003', '17226004', '17225980', '17225981', '17225982', '17225983', '17225984', '96275'] proc_lsst.multi INFO: new jobs since last cancel ['17226005', '17226006', '17226007', '17226008', '17226009', '17226010', '17225950', '17225951', '17225952', '17225953', '17225954', '17225955', '17225956', '17225957', '17225958', '17225959', '17225960', '17225961', '17225962', '17225963', '17225964', '17225965', '17225966', '17225967', '17225968', '17225969', '17225970', '17225971', '17225972', '17225973', '17225974', '17225975', '17225976', '17225977', '17225978', '17225979', '17225985', '17225986', '17225987', '17225988', '17225989', '17225990', '17225991', '17225992', '17225993', '17225994', '17225995', '17225996', '17225997', '17225998', '17225999', '17226000', '17226001', '17226002', '17226003', '17226004', '17225980', '17225981', '17225982', '17225983', '17225984', '96275'] 2024-03-19 00:41:23 proc_lsst.multi:146 [INFO] found job 17226005 in provider checkpoint proc_lsst.multi INFO: found job 17226005 in provider checkpoint 2024-03-19 00:41:23 proc_lsst.multi:201 [INFO] cancelling 17226005 on provider checkpoint proc_lsst.multi INFO: cancelling 17226005 on provider checkpoint 2024-03-19 00:41:23 proc_lsst.multi:146 [INFO] found job 17226006 in provider checkpoint proc_lsst.multi INFO: found job 17226006 in provider checkpoint 2024-03-19 00:41:23 proc_lsst.multi:201 [INFO] cancelling 17226006 on provider checkpoint proc_lsst.multi INFO: cancelling 17226006 on provider checkpoint 2024-03-19 00:41:23 proc_lsst.multi:146 [INFO] found job 17226007 in provider checkpoint proc_lsst.multi INFO: found job 17226007 in provider checkpoint 2024-03-19 00:41:23 proc_lsst.multi:201 [INFO] cancelling 17226007 on provider checkpoint proc_lsst.multi INFO: cancelling 17226007 on provider checkpoint 2024-03-19 00:41:24 proc_lsst.multi:146 [INFO] found job 17226008 in provider checkpoint proc_lsst.multi INFO: found job 17226008 in provider checkpoint 2024-03-19 00:41:24 proc_lsst.multi:201 [INFO] cancelling 17226008 on provider checkpoint proc_lsst.multi INFO: cancelling 17226008 on provider checkpoint 2024-03-19 00:41:24 proc_lsst.multi:146 [INFO] found job 17226009 in provider checkpoint proc_lsst.multi INFO: found job 17226009 in provider checkpoint 2024-03-19 00:41:24 proc_lsst.multi:201 [INFO] cancelling 17226009 on provider checkpoint proc_lsst.multi INFO: cancelling 17226009 on provider checkpoint 2024-03-19 00:41:24 proc_lsst.multi:146 [INFO] found job 17226010 in provider checkpoint proc_lsst.multi INFO: found job 17226010 in provider checkpoint 2024-03-19 00:41:24 proc_lsst.multi:201 [INFO] cancelling 17226010 on provider checkpoint proc_lsst.multi INFO: cancelling 17226010 on provider checkpoint 2024-03-19 00:41:24 proc_lsst.multi:146 [INFO] found job 17225950 in provider astro proc_lsst.multi INFO: found job 17225950 in provider astro 2024-03-19 00:41:24 proc_lsst.multi:201 [INFO] cancelling 17225950 on provider astro proc_lsst.multi INFO: cancelling 17225950 on provider astro 2024-03-19 00:41:24 proc_lsst.multi:146 [INFO] found job 17225951 in provider astro proc_lsst.multi INFO: found job 17225951 in provider astro 2024-03-19 00:41:24 proc_lsst.multi:201 [INFO] cancelling 17225951 on provider astro proc_lsst.multi INFO: cancelling 17225951 on provider astro 2024-03-19 00:41:24 proc_lsst.multi:146 [INFO] found job 17225952 in provider astro proc_lsst.multi INFO: found job 17225952 in provider astro 2024-03-19 00:41:24 proc_lsst.multi:201 [INFO] cancelling 17225952 on provider astro proc_lsst.multi INFO: cancelling 17225952 on provider astro 2024-03-19 00:41:24 proc_lsst.multi:146 [INFO] found job 17225953 in provider astro proc_lsst.multi INFO: found job 17225953 in provider astro 2024-03-19 00:41:24 proc_lsst.multi:201 [INFO] cancelling 17225953 on provider astro proc_lsst.multi INFO: cancelling 17225953 on provider astro 2024-03-19 00:41:25 proc_lsst.multi:146 [INFO] found job 17225954 in provider astro proc_lsst.multi INFO: found job 17225954 in provider astro 2024-03-19 00:41:25 proc_lsst.multi:201 [INFO] cancelling 17225954 on provider astro proc_lsst.multi INFO: cancelling 17225954 on provider astro 2024-03-19 00:41:25 proc_lsst.multi:146 [INFO] found job 17225955 in provider astro proc_lsst.multi INFO: found job 17225955 in provider astro 2024-03-19 00:41:25 proc_lsst.multi:201 [INFO] cancelling 17225955 on provider astro proc_lsst.multi INFO: cancelling 17225955 on provider astro 2024-03-19 00:41:25 proc_lsst.multi:146 [INFO] found job 17225956 in provider astro proc_lsst.multi INFO: found job 17225956 in provider astro 2024-03-19 00:41:25 proc_lsst.multi:201 [INFO] cancelling 17225956 on provider astro proc_lsst.multi INFO: cancelling 17225956 on provider astro 2024-03-19 00:41:25 proc_lsst.multi:146 [INFO] found job 17225957 in provider astro proc_lsst.multi INFO: found job 17225957 in provider astro 2024-03-19 00:41:25 proc_lsst.multi:201 [INFO] cancelling 17225957 on provider astro proc_lsst.multi INFO: cancelling 17225957 on provider astro 2024-03-19 00:41:25 proc_lsst.multi:146 [INFO] found job 17225958 in provider astro proc_lsst.multi INFO: found job 17225958 in provider astro 2024-03-19 00:41:25 proc_lsst.multi:201 [INFO] cancelling 17225958 on provider astro proc_lsst.multi INFO: cancelling 17225958 on provider astro 2024-03-19 00:41:25 proc_lsst.multi:146 [INFO] found job 17225959 in provider astro proc_lsst.multi INFO: found job 17225959 in provider astro 2024-03-19 00:41:25 proc_lsst.multi:201 [INFO] cancelling 17225959 on provider astro proc_lsst.multi INFO: cancelling 17225959 on provider astro 2024-03-19 00:41:25 proc_lsst.multi:146 [INFO] found job 17225960 in provider astro proc_lsst.multi INFO: found job 17225960 in provider astro 2024-03-19 00:41:25 proc_lsst.multi:201 [INFO] cancelling 17225960 on provider astro proc_lsst.multi INFO: cancelling 17225960 on provider astro 2024-03-19 00:41:25 proc_lsst.multi:146 [INFO] found job 17225961 in provider astro proc_lsst.multi INFO: found job 17225961 in provider astro 2024-03-19 00:41:25 proc_lsst.multi:201 [INFO] cancelling 17225961 on provider astro proc_lsst.multi INFO: cancelling 17225961 on provider astro 2024-03-19 00:41:25 proc_lsst.multi:146 [INFO] found job 17225962 in provider astro proc_lsst.multi INFO: found job 17225962 in provider astro 2024-03-19 00:41:25 proc_lsst.multi:201 [INFO] cancelling 17225962 on provider astro proc_lsst.multi INFO: cancelling 17225962 on provider astro 2024-03-19 00:41:26 proc_lsst.multi:146 [INFO] found job 17225963 in provider astro proc_lsst.multi INFO: found job 17225963 in provider astro 2024-03-19 00:41:26 proc_lsst.multi:201 [INFO] cancelling 17225963 on provider astro proc_lsst.multi INFO: cancelling 17225963 on provider astro 2024-03-19 00:41:26 proc_lsst.multi:146 [INFO] found job 17225964 in provider astro proc_lsst.multi INFO: found job 17225964 in provider astro 2024-03-19 00:41:26 proc_lsst.multi:201 [INFO] cancelling 17225964 on provider astro proc_lsst.multi INFO: cancelling 17225964 on provider astro 2024-03-19 00:41:26 proc_lsst.multi:146 [INFO] found job 17225965 in provider astro proc_lsst.multi INFO: found job 17225965 in provider astro 2024-03-19 00:41:26 proc_lsst.multi:201 [INFO] cancelling 17225965 on provider astro proc_lsst.multi INFO: cancelling 17225965 on provider astro 2024-03-19 00:41:26 proc_lsst.multi:146 [INFO] found job 17225966 in provider astro proc_lsst.multi INFO: found job 17225966 in provider astro 2024-03-19 00:41:26 proc_lsst.multi:201 [INFO] cancelling 17225966 on provider astro proc_lsst.multi INFO: cancelling 17225966 on provider astro 2024-03-19 00:41:26 proc_lsst.multi:146 [INFO] found job 17225967 in provider astro proc_lsst.multi INFO: found job 17225967 in provider astro 2024-03-19 00:41:26 proc_lsst.multi:201 [INFO] cancelling 17225967 on provider astro proc_lsst.multi INFO: cancelling 17225967 on provider astro 2024-03-19 00:41:26 proc_lsst.multi:146 [INFO] found job 17225968 in provider astro proc_lsst.multi INFO: found job 17225968 in provider astro 2024-03-19 00:41:26 proc_lsst.multi:201 [INFO] cancelling 17225968 on provider astro proc_lsst.multi INFO: cancelling 17225968 on provider astro 2024-03-19 00:41:26 proc_lsst.multi:146 [INFO] found job 17225969 in provider astro proc_lsst.multi INFO: found job 17225969 in provider astro 2024-03-19 00:41:26 proc_lsst.multi:201 [INFO] cancelling 17225969 on provider astro proc_lsst.multi INFO: cancelling 17225969 on provider astro 2024-03-19 00:41:26 proc_lsst.multi:146 [INFO] found job 17225970 in provider astro proc_lsst.multi INFO: found job 17225970 in provider astro 2024-03-19 00:41:26 proc_lsst.multi:201 [INFO] cancelling 17225970 on provider astro proc_lsst.multi INFO: cancelling 17225970 on provider astro 2024-03-19 00:41:27 proc_lsst.multi:146 [INFO] found job 17225971 in provider astro proc_lsst.multi INFO: found job 17225971 in provider astro 2024-03-19 00:41:27 proc_lsst.multi:201 [INFO] cancelling 17225971 on provider astro proc_lsst.multi INFO: cancelling 17225971 on provider astro 2024-03-19 00:41:27 proc_lsst.multi:146 [INFO] found job 17225972 in provider astro proc_lsst.multi INFO: found job 17225972 in provider astro 2024-03-19 00:41:27 proc_lsst.multi:201 [INFO] cancelling 17225972 on provider astro proc_lsst.multi INFO: cancelling 17225972 on provider astro 2024-03-19 00:41:27 proc_lsst.multi:146 [INFO] found job 17225973 in provider astro proc_lsst.multi INFO: found job 17225973 in provider astro 2024-03-19 00:41:27 proc_lsst.multi:201 [INFO] cancelling 17225973 on provider astro proc_lsst.multi INFO: cancelling 17225973 on provider astro 2024-03-19 00:41:27 proc_lsst.multi:146 [INFO] found job 17225974 in provider astro proc_lsst.multi INFO: found job 17225974 in provider astro 2024-03-19 00:41:27 proc_lsst.multi:201 [INFO] cancelling 17225974 on provider astro proc_lsst.multi INFO: cancelling 17225974 on provider astro 2024-03-19 00:41:27 proc_lsst.multi:146 [INFO] found job 17225975 in provider astro proc_lsst.multi INFO: found job 17225975 in provider astro 2024-03-19 00:41:27 proc_lsst.multi:201 [INFO] cancelling 17225975 on provider astro proc_lsst.multi INFO: cancelling 17225975 on provider astro 2024-03-19 00:41:27 proc_lsst.multi:146 [INFO] found job 17225976 in provider astro proc_lsst.multi INFO: found job 17225976 in provider astro 2024-03-19 00:41:27 proc_lsst.multi:201 [INFO] cancelling 17225976 on provider astro proc_lsst.multi INFO: cancelling 17225976 on provider astro 2024-03-19 00:41:27 proc_lsst.multi:146 [INFO] found job 17225977 in provider astro proc_lsst.multi INFO: found job 17225977 in provider astro 2024-03-19 00:41:27 proc_lsst.multi:201 [INFO] cancelling 17225977 on provider astro proc_lsst.multi INFO: cancelling 17225977 on provider astro 2024-03-19 00:41:27 proc_lsst.multi:146 [INFO] found job 17225978 in provider astro proc_lsst.multi INFO: found job 17225978 in provider astro 2024-03-19 00:41:27 proc_lsst.multi:201 [INFO] cancelling 17225978 on provider astro proc_lsst.multi INFO: cancelling 17225978 on provider astro 2024-03-19 00:41:27 proc_lsst.multi:146 [INFO] found job 17225979 in provider astro proc_lsst.multi INFO: found job 17225979 in provider astro 2024-03-19 00:41:27 proc_lsst.multi:201 [INFO] cancelling 17225979 on provider astro proc_lsst.multi INFO: cancelling 17225979 on provider astro 2024-03-19 00:41:27 proc_lsst.multi:146 [INFO] found job 17225985 in provider escience_a40 proc_lsst.multi INFO: found job 17225985 in provider escience_a40 2024-03-19 00:41:27 proc_lsst.multi:201 [INFO] cancelling 17225985 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225985 on provider escience_a40 2024-03-19 00:41:28 proc_lsst.multi:146 [INFO] found job 17225986 in provider escience_a40 proc_lsst.multi INFO: found job 17225986 in provider escience_a40 2024-03-19 00:41:28 proc_lsst.multi:201 [INFO] cancelling 17225986 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225986 on provider escience_a40 2024-03-19 00:41:28 proc_lsst.multi:146 [INFO] found job 17225987 in provider escience_a40 proc_lsst.multi INFO: found job 17225987 in provider escience_a40 2024-03-19 00:41:28 proc_lsst.multi:201 [INFO] cancelling 17225987 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225987 on provider escience_a40 2024-03-19 00:41:28 proc_lsst.multi:146 [INFO] found job 17225988 in provider escience_a40 proc_lsst.multi INFO: found job 17225988 in provider escience_a40 2024-03-19 00:41:28 proc_lsst.multi:201 [INFO] cancelling 17225988 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225988 on provider escience_a40 2024-03-19 00:41:28 proc_lsst.multi:146 [INFO] found job 17225989 in provider escience_a40 proc_lsst.multi INFO: found job 17225989 in provider escience_a40 2024-03-19 00:41:28 proc_lsst.multi:201 [INFO] cancelling 17225989 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225989 on provider escience_a40 2024-03-19 00:41:28 proc_lsst.multi:146 [INFO] found job 17225990 in provider escience_a40 proc_lsst.multi INFO: found job 17225990 in provider escience_a40 2024-03-19 00:41:28 proc_lsst.multi:201 [INFO] cancelling 17225990 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225990 on provider escience_a40 2024-03-19 00:41:28 proc_lsst.multi:146 [INFO] found job 17225991 in provider escience_a40 proc_lsst.multi INFO: found job 17225991 in provider escience_a40 2024-03-19 00:41:28 proc_lsst.multi:201 [INFO] cancelling 17225991 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225991 on provider escience_a40 2024-03-19 00:41:28 proc_lsst.multi:146 [INFO] found job 17225992 in provider escience_a40 proc_lsst.multi INFO: found job 17225992 in provider escience_a40 2024-03-19 00:41:28 proc_lsst.multi:201 [INFO] cancelling 17225992 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225992 on provider escience_a40 2024-03-19 00:41:28 proc_lsst.multi:146 [INFO] found job 17225993 in provider escience_a40 proc_lsst.multi INFO: found job 17225993 in provider escience_a40 2024-03-19 00:41:28 proc_lsst.multi:201 [INFO] cancelling 17225993 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225993 on provider escience_a40 2024-03-19 00:41:28 proc_lsst.multi:146 [INFO] found job 17225994 in provider escience_a40 proc_lsst.multi INFO: found job 17225994 in provider escience_a40 2024-03-19 00:41:28 proc_lsst.multi:201 [INFO] cancelling 17225994 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225994 on provider escience_a40 2024-03-19 00:41:28 proc_lsst.multi:146 [INFO] found job 17225995 in provider escience_a40 proc_lsst.multi INFO: found job 17225995 in provider escience_a40 2024-03-19 00:41:28 proc_lsst.multi:201 [INFO] cancelling 17225995 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225995 on provider escience_a40 2024-03-19 00:41:28 proc_lsst.multi:146 [INFO] found job 17225996 in provider escience_a40 proc_lsst.multi INFO: found job 17225996 in provider escience_a40 2024-03-19 00:41:28 proc_lsst.multi:201 [INFO] cancelling 17225996 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225996 on provider escience_a40 2024-03-19 00:41:29 proc_lsst.multi:146 [INFO] found job 17225997 in provider escience_a40 proc_lsst.multi INFO: found job 17225997 in provider escience_a40 2024-03-19 00:41:29 proc_lsst.multi:201 [INFO] cancelling 17225997 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225997 on provider escience_a40 2024-03-19 00:41:29 proc_lsst.multi:146 [INFO] found job 17225998 in provider escience_a40 proc_lsst.multi INFO: found job 17225998 in provider escience_a40 2024-03-19 00:41:29 proc_lsst.multi:201 [INFO] cancelling 17225998 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225998 on provider escience_a40 2024-03-19 00:41:29 proc_lsst.multi:146 [INFO] found job 17225999 in provider escience_a40 proc_lsst.multi INFO: found job 17225999 in provider escience_a40 2024-03-19 00:41:29 proc_lsst.multi:201 [INFO] cancelling 17225999 on provider escience_a40 proc_lsst.multi INFO: cancelling 17225999 on provider escience_a40 2024-03-19 00:41:29 proc_lsst.multi:146 [INFO] found job 17226000 in provider escience_a40 proc_lsst.multi INFO: found job 17226000 in provider escience_a40 2024-03-19 00:41:29 proc_lsst.multi:201 [INFO] cancelling 17226000 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226000 on provider escience_a40 2024-03-19 00:41:29 proc_lsst.multi:146 [INFO] found job 17226001 in provider escience_a40 proc_lsst.multi INFO: found job 17226001 in provider escience_a40 2024-03-19 00:41:29 proc_lsst.multi:201 [INFO] cancelling 17226001 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226001 on provider escience_a40 2024-03-19 00:41:29 proc_lsst.multi:146 [INFO] found job 17226002 in provider escience_a40 proc_lsst.multi INFO: found job 17226002 in provider escience_a40 2024-03-19 00:41:29 proc_lsst.multi:201 [INFO] cancelling 17226002 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226002 on provider escience_a40 2024-03-19 00:41:29 proc_lsst.multi:146 [INFO] found job 17226003 in provider escience_a40 proc_lsst.multi INFO: found job 17226003 in provider escience_a40 2024-03-19 00:41:29 proc_lsst.multi:201 [INFO] cancelling 17226003 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226003 on provider escience_a40 2024-03-19 00:41:29 proc_lsst.multi:146 [INFO] found job 17226004 in provider escience_a40 proc_lsst.multi INFO: found job 17226004 in provider escience_a40 2024-03-19 00:41:29 proc_lsst.multi:201 [INFO] cancelling 17226004 on provider escience_a40 proc_lsst.multi INFO: cancelling 17226004 on provider escience_a40 2024-03-19 00:41:29 proc_lsst.multi:146 [INFO] found job 17225980 in provider escience_rtx6k proc_lsst.multi INFO: found job 17225980 in provider escience_rtx6k 2024-03-19 00:41:29 proc_lsst.multi:201 [INFO] cancelling 17225980 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17225980 on provider escience_rtx6k 2024-03-19 00:41:29 proc_lsst.multi:146 [INFO] found job 17225981 in provider escience_rtx6k proc_lsst.multi INFO: found job 17225981 in provider escience_rtx6k 2024-03-19 00:41:29 proc_lsst.multi:201 [INFO] cancelling 17225981 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17225981 on provider escience_rtx6k 2024-03-19 00:41:29 proc_lsst.multi:146 [INFO] found job 17225982 in provider escience_rtx6k proc_lsst.multi INFO: found job 17225982 in provider escience_rtx6k 2024-03-19 00:41:29 proc_lsst.multi:201 [INFO] cancelling 17225982 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17225982 on provider escience_rtx6k 2024-03-19 00:41:29 proc_lsst.multi:146 [INFO] found job 17225983 in provider escience_rtx6k proc_lsst.multi INFO: found job 17225983 in provider escience_rtx6k 2024-03-19 00:41:29 proc_lsst.multi:201 [INFO] cancelling 17225983 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17225983 on provider escience_rtx6k 2024-03-19 00:41:29 proc_lsst.multi:146 [INFO] found job 17225984 in provider escience_rtx6k proc_lsst.multi INFO: found job 17225984 in provider escience_rtx6k 2024-03-19 00:41:29 proc_lsst.multi:201 [INFO] cancelling 17225984 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17225984 on provider escience_rtx6k 2024-03-19 00:41:30 proc_lsst.multi:146 [INFO] found job 96275 in provider local proc_lsst.multi INFO: found job 96275 in provider local 2024-03-19 00:41:30 proc_lsst.multi:201 [INFO] cancelling 96275 on provider local proc_lsst.multi INFO: cancelling 96275 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 96275 parsl.providers.local.local WARNING: Failed to kill PID: 96275 and child processes on local 2024-03-19 00:41:31 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 148.3991 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20220527_A0b_science#step0_20240319T003842Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 148.4061 seconds; current memory usage: 0.275 Gibyte, delta: 0.001 Gibyte, peak delta: 0.006 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 170.8883 seconds; current memory usage: 0.275 Gibyte, delta: 0.138 Gibyte, peak delta: 0.144 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.280 Gibyte (main), 0.280 Gibyte (largest child process) Run Id: None Run Name: DEEP_20220527_A0b_science#step0_20240319T003842Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 2m52.801s user 0m36.504s sys 0m18.500s